A log inspection method, device and electronic device based on machine learning

Abstract

The present invention relates to the field of computer technology, and more particularly to a log inspection method, device, and electronic device based on machine learning, comprising: obtaining a log to be inspected, screening the log to be inspected, and obtaining an abnormal log; matching the abnormal log with an abnormal log template to obtain an abnormal log template corresponding to the abnormal log; extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set; inputting the abnormal log feature set into an abnormal log model to obtain a detection result of the abnormal log; obtaining an analysis result of the detection result, feeding the analysis result back to the abnormal log model, and updating the abnormal log model; creating a tracking task, and tracking and inspecting the abnormal log. The present invention achieves tracking of abnormal logs by automatically creating tracking tasks, freeing up developers, and continuously improving the inspection accuracy of abnormal logs by automatically updating the model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log inspection method, device, and electronic device based on machine learning. Background Art

[0002] The modern Internet is basically a microservice architecture, with many systems and a large workload for log inspection. This compresses product development time, resulting in reduced product quality. Reduced product quality in turn increases the workload for log inspection, forming a vicious cycle.

[0003] In order not to reduce the quality of the products produced, the product development time will inevitably be extended, which will reduce the competitiveness of the company's products in the long run. At the same time, the inspection effect is heavily dependent on the developers, such as Figure 1 As shown, in the existing inspection process, the abnormal log inspection, abnormal cause analysis, manual recording, task tracking creation, and information synchronization stages all require the participation of developers, and the self-consciousness of developers has a great impact on the inspection effect.

[0004] On the other hand, due to security restrictions, production data cannot be downloaded directly to the local computer, which increases the workload and affects the work experience. Summary of the Invention

[0005] In response to the above technical problems, the present invention provides a log inspection method, device and electronic device based on machine learning to achieve the tracking of abnormal logs, liberate developers, and continuously improve the inspection accuracy of abnormal logs to protect data security.

[0006] The embodiments of this specification provide a log inspection method based on machine learning, including:

[0007] Obtaining logs to be inspected, screening the logs to be inspected, and obtaining abnormal logs;

[0008] Performing exception log template matching on the exception log to obtain an exception log template corresponding to the exception log;

[0009] Extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0010] Inputting the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log;

[0011] Obtaining an analysis result of the detection result, feeding the analysis result back to the abnormality log model, and updating the abnormality log model;

[0012] Create a tracking task to track and inspect the abnormal log.

[0013] Preferably, before obtaining the log to be inspected, the following steps are performed:

[0014] Scan historical logs and extract historical abnormal log sample sets;

[0015] Parsing the historical abnormal log sample set using regular expressions to obtain an abnormal log template;

[0016] Assigning a unique identity to the abnormal log template and classifying the abnormal log template by importance according to the weight;

[0017] Extracting a historical abnormal log feature set of the historical abnormal log sample set based on the abnormal log template, and performing feature processing on the historical abnormal log feature set;

[0018] The processed historical abnormal log feature set is input into the deep neural network model training to obtain the abnormal log model.

[0019] Preferably, obtaining the log to be inspected includes:

[0020] The directory where the logs are written is monitored in real time, and the written logs are transmitted to a message subscription center in the form of messages to obtain the logs to be inspected.

[0021] Preferably, the extracting the historical abnormal log feature set of the historical abnormal log sample set based on the abnormal log template includes:

[0022] Performing statistics on the historical abnormal log sample set according to the unique identity identifier of the abnormal log template, and performing mean and variance processing in combination with the weight of the abnormal log template to obtain statistical features; and / or,

[0023] Obtaining a comparison feature by comparing the values ​​before and after the sequence in the historical abnormal log sample set; and / or,

[0024] Introducing contextual features;

[0025] The statistical features and / or the comparison features and / or the context features are combined to obtain an abnormal log feature set of the historical abnormal log sample set.

[0026] Preferably, after extracting the historical abnormal log sample set, the following steps are included:

[0027] Oversampling is performed on the historical abnormal log sample set.

[0028] Preferably, the feature processing of the historical abnormal log feature set includes:

[0029] Encoding the discrete features in the historical abnormal log feature set using a single-bit effective code to obtain a binary feature;

[0030] Feature scaling is performed on the historical abnormal log feature set to obtain the processed historical abnormal log feature set.

[0031] Preferably, obtaining the comparative features by comparing the values ​​before and after the sequence in the historical abnormal log sample set includes:

[0032] The comparison feature is obtained by comparing the values ​​before and after the sequence in the historical log sample set using a difference and / or ratio comparison method.

[0033] Preferably, obtaining the detection result of the abnormal log includes:

[0034] Using F score as the evaluation indicator, the optimal threshold is determined through grid search algorithm and k-fold cross validation;

[0035] The output result of the abnormal log model is binary-classified according to the optimal threshold to obtain the detection result of the abnormal log.

[0036] The embodiments of this specification also provide a log inspection device based on machine learning, including:

[0037] The abnormal log acquisition module is used to obtain the logs to be inspected, filter the logs to be inspected, and obtain abnormal logs;

[0038] A template matching module is used to perform abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log;

[0039] A feature extraction module, configured to extract features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0040] An abnormal log detection module is used to input the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log;

[0041] The model updating module is used to obtain an analysis result of the detection result, feed the analysis result back to the abnormality log model, and update the abnormality log model.

[0042] The tracking module is used to create a tracking task and track and inspect the abnormal log.

[0043] An electronic device, wherein the electronic device comprises:

[0044] A processor and a memory storing a computer-executable program, wherein when the executable program is executed, the processor is caused to perform any of the above methods.

[0045] A computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the one or more programs implement any of the above methods.

[0046] The present invention realizes the tracking of exception logs by automatically creating tracking tasks, freeing up developers. At the same time, the model can be automatically updated according to the feedback information, continuously improving the model's inspection accuracy for exception logs and protecting data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 This is a log inspection flow chart of the existing technology;

[0049] Figure 2 A hardware structure diagram of a log inspection mobile terminal based on machine learning provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the principle of a log inspection method based on machine learning provided in an embodiment of this specification;

[0051] Figure 4 A flowchart of a log inspection process based on machine learning provided in an embodiment of this specification;

[0052] Figure 5 A flow chart of log collection provided in the embodiments of this specification;

[0053] Figure 6 A flow chart for generating an abnormal log model provided in the embodiments of this specification;

[0054] Figure 7 A neural network layer diagram within the deep neural network model provided in the embodiments of this specification;

[0055] Figure 8 This is a schematic diagram of the structure of a log inspection device based on machine learning provided in an embodiment of this specification. DETAILED DESCRIPTION

[0056] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in various forms, and it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, making it easier to fully convey the inventive concept to those skilled in the art. In the figures, the same reference numerals represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0057] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0058] In the description of specific embodiments, the features, structures, characteristics, or other details of the present invention are described to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from practicing the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0059] The figures shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be performed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0060] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0061] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0062] Figure 1 The log inspection flow chart of the existing technology shows that the abnormal log inspection, abnormal cause analysis, manual recording, task tracking creation, and information synchronization stages all require the participation of developers. The inspection effect is heavily dependent on the developers, and the developers' self-consciousness has a great impact on the inspection effect. Therefore, the present invention proposes a log inspection method based on machine learning, such as Figure 4 As shown in the figure, when inspecting the exception log, the developer's participation is only required in analyzing the cause of the exception, reporting true or false exceptions, and whether tracking is required, as well as information synchronization. Figure 1Compared with the existing technology shown, the amount of developer participation is reduced. At the same time, the present invention uses feedback of true and false exceptions to feed back to the daily log model in exception log screening, continuously improving the inspection accuracy. In terms of exception log tracking, tasks can be automatically created according to the feedback results to track exception logs, liberating developers and realizing data security protection.

[0063] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a mobile terminal as an example, Figure 2 This is a hardware structure diagram of a log inspection mobile terminal based on machine learning. Figure 2 As shown, the mobile terminal may include one or more ( Figure 2 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 2 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown.

[0064] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the page layout method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the mobile terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0066] Reference Figure 3 A schematic diagram of the principle of a log inspection method based on machine learning provided in an embodiment of this specification includes:

[0067] S201: Obtain logs to be inspected, filter the logs to be inspected, and obtain abnormal logs;

[0068] In a preferred embodiment of the present invention, Figure 5 As shown, the log directory on the disk is monitored in real time by the log collection system. When the application has new logs written to the log directory on the disk, the log collection system collects the logs written to the disk and passes the written logs to the distributed publish-subscribe messaging system in the form of messages. The computing engine processes the messages in real time and stores the written logs in the distributed file system through the distributed publish-subscribe messaging system for offline analysis. The above method realizes the monitoring and real-time processing of logs, and at the same time uses the data in the distributed file system for analysis to improve the processing effect of logs. In this embodiment, optionally, the log collection system uses Flume, the distributed publish-subscribe messaging system uses Kafka, the computing engine uses Spark Streaming, and the distributed file system uses HDFS. The Flume+Kafka+Spark method is only used as the optimal description, and no specific restrictions are made on the above systems or engines.

[0069] S202: performing abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log;

[0070] In a preferred embodiment of the present invention, log template matching is performed on the exception log through regular expressions to obtain the exception log template corresponding to the exception log, and then the unique identity identifier corresponding to the matched exception log template is queried from the cache to provide services for the subsequent extraction of the exception log feature set.

[0071] S203: Extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0072] In a preferred embodiment of the present invention, due to the high real-time nature of anomaly detection, stream processing is used for feature extraction to avoid timeouts. Taking into account environmental and seasonal factors, and incorporating the characteristics of time series data, different window widths are selected when extracting the anomaly log feature set. Consequently, the anomaly log feature set includes a cross-combination of window widths, statistical features, and comparison features, as well as contextual features. Finally, the processed data is written to a Hive table using a Spark module for processing structured data, for subsequent use by the anomaly log model.

[0073] S204: Inputting the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log;

[0074] In a preferred embodiment of the present invention, a Spark module for processing structured data retrieves anomaly log feature sets from a Hive table. This is then used to detect anomaly logs using an anomaly log model, yielding detection results. However, some anomaly patterns exist only in the anomaly log test set, and the anomaly log model cannot learn these patterns from the anomaly log training set. For these anomalies, integrated business rules are used for identification, such as monitoring. This approach eliminates the need for extensive human intervention in detecting anomaly logs.

[0075] S205: Obtain analysis results of the detection results, feed the analysis results back to the abnormality log model, and update the abnormality log model;

[0076] In a preferred embodiment of the present invention, Figure 4 As shown in the figure, after obtaining the detection results of the exception log, the system displays the request serial number and the error location in the code. Then the developer analyzes the cause of the error based on the given error information and feeds back to the exception log model, so that the exception log model can be continuously updated and the inspection accuracy can be continuously improved. Among them, it is best to update the exception log model once a week.

[0077] S206: Create a tracking task to track and inspect the abnormal log.

[0078] In a preferred embodiment of the present invention, based on the feedback from the above-mentioned developers, the system automatically creates tasks to track exception logs, freeing up developers and avoiding excessive interference in exception log inspections caused by developers' involvement. At the same time, it achieves security protection for data and prevents the phenomenon of production data being unable to be downloaded due to security restrictions.

[0079] Furthermore, before obtaining the log to be inspected, the following steps are performed:

[0080] Scan historical logs and extract historical abnormal log sample sets;

[0081] Parsing the historical abnormal log sample set using regular expressions to obtain an abnormal log template;

[0082] Assigning a unique identity to the abnormal log template and classifying the abnormal log template by importance according to the weight;

[0083] Extracting a historical abnormal log feature set of the historical abnormal log sample set based on the abnormal log template, and performing feature processing on the historical abnormal log feature set;

[0084] The processed historical abnormal log feature set is input into the deep neural network model training to obtain the abnormal log model.

[0085] In a preferred embodiment of the present invention, referring to Figure 6 , scan the code in the historical log, extract the historical exception log sample set, and then parse the historical exception log sample set through regular expressions to obtain the exception log template. After extracting the log template content, a unique identity will be assigned and saved in the database. Although they are all exception log templates, their importance is different. Therefore, here the exception log templates are divided into 5 levels according to their importance. The higher the level, the greater the weight. Sometimes, there will be some unexpected situations, such as fixing online vulnerabilities. For such situations, you can also manually add exception log templates and weights in the background management. Then, based on the exception log template, the historical exception log feature set of the historical exception log sample set is extracted, and the historical exception log feature set is processed. The processed historical exception log feature set is input into the deep neural network model training using supervised learning to obtain the exception log model. The exception log model is trained in the above way, and then it can be verified through the exception log sample verification set. The exception log model with the best detection effect is taken as the model for subsequent exception log inspections. Among them, before extracting the historical abnormal log feature set of the historical abnormal log sample set, the historical abnormal log sample set should be preprocessed to ensure the accuracy of the abnormal log model in detecting abnormal logs. At the same time, since time series data has a strong correlation with time, future data should be avoided during training and verification.

[0086] Reference Figure 7The neural network layer diagram inside the deep neural network model provided in the embodiment of this specification, the neural network layers inside the deep neural network model can be divided into three categories, input layer, hidden layer and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. There can be multiple hidden layers to enhance the expressive power of the model. Here, two fully connected hidden layers and one Sigmoid output layer are used. The number of hidden neurons in both layers is 128, and the activation function uses a linear rectifier function. Dropout and L1 regularization are used to control the overfitting of the abnormal log model to enhance the generalization ability.

[0087] Furthermore, obtaining the log to be inspected includes:

[0088] The directory where the logs are written is monitored in real time, and the written logs are transmitted to a message subscription center in the form of messages to obtain the logs to be inspected.

[0089] Furthermore, the extracting of the historical abnormal log feature set from the historical abnormal log sample set based on the abnormal log template includes:

[0090] Performing statistics on the historical abnormal log sample set according to the unique identity identifier of the abnormal log template, and performing mean and variance processing in combination with the weight of the abnormal log template to obtain statistical features; and / or,

[0091] Obtaining a comparison feature by comparing the values ​​before and after the sequence in the historical abnormal log sample set; and / or,

[0092] Introducing contextual features;

[0093] The statistical features and / or the comparison features and / or the context features are combined to obtain an abnormal log feature set of the historical abnormal log sample set.

[0094] In a preferred embodiment of the present invention, a sliding window is used to extract statistical features of window-type data. Specifically, Spark Streaming is used to collect statistics on historical abnormal logs according to their unique identity identifiers, and then the mean and variance are processed in combination with the weight of the abnormal log template. The mean and variance processing formulas are as follows:

[0095] Mean:

[0096]

[0097] variance:

[0098]

[0099] Where n is the number of unique identifiers, is the characteristic mean, x is the characteristic value, i represents the i-th characteristic value, S 2 is the characteristic variance.

[0100] The statistical features of the historical abnormal log sample set are extracted in the above method. Then, by comparing the values ​​before and after the sequence in the historical abnormal log sample set, the comparative features are obtained. Then, contextual features such as date and UV (Unique Visitor) are introduced to finally form the matrix shown in Table 1. Table 1 is as follows:

[0101]

[0102] Table 1

[0103] Furthermore, after extracting the historical abnormal log sample set, it includes:

[0104] Oversampling is performed on the historical abnormal log sample set.

[0105] In a preferred embodiment of the present invention, in the system, the proportion of truly abnormal log samples is less than 5%, which is much lower than that of normal samples. If the historical abnormal log sample set is directly trained, the abnormal log model will tend to predict the samples as normal, and the purpose of abnormality detection will not be achieved. In order to solve this problem, the historical abnormal log sample set is processed by oversampling to achieve a 1:1 ratio of positive and negative samples.

[0106] Furthermore, the feature processing of the historical abnormal log feature set includes:

[0107] Encoding the discrete features in the historical abnormal log feature set using a single-bit effective code to obtain a binary feature;

[0108] Feature scaling is performed on the historical abnormal log feature set to obtain the processed historical abnormal log feature set.

[0109] In a preferred embodiment of the present invention, discrete features in the historical anomaly log feature set are encoded using one-hot encoding, converting the discrete features into binary features. For example, the dates in Table 1 are processed into three states: weekdays, holidays, and other. Table 1 shows that the differences between different features are significant. If left unaddressed, the "large number eats the small number" phenomenon can easily occur. This also affects the convergence of the algorithm, causing it to fail to converge or to converge with great difficulty. When using a neural network solved by gradient descent, this can also cause gradient explosion. Therefore, feature scaling can be performed using normalization or standardization.

[0110] Normalization formula:

[0111]

[0112] Normalization formula:

[0113]

[0114] Among them, x is the eigenvalue, x′ is the feature scaling value, u represents the feature mean, and σ represents the feature standard deviation.

[0115] Max(x) and min(x) in the normalization formula are difficult to determine and need to be specified based on empirical values. Therefore, it is best to use a standardized approach to perform feature scaling on the historical anomaly log feature set.

[0116] Furthermore, the comparison features are obtained by comparing the values ​​before and after the sequence in the historical abnormal log sample set, including:

[0117] The comparison feature is obtained by comparing the values ​​before and after the sequence in the historical log sample set using a difference and / or ratio comparison method.

[0118] In a preferred embodiment of the present invention, two comparison methods, difference and ratio, are used. Difference represents absolute change, while ratio represents relative value.

[0119] difference:

[0120] ΔX T =X T -X T-1

[0121] Proportion:

[0122]

[0123] Where ΔX T is the difference between features, X T is the eigenvector, X T-1 ≥ threshold, it should be noted that when X T-1 When it is close to 0, it is easy to be distorted, so a threshold is set. T-1 When the threshold is less than the value, the experience value will be used.

[0124] By comparing the values ​​before and after the sequence in the historical log sample set in the above way, the comparison features are obtained.

[0125] Furthermore, obtaining the detection result of the abnormal log includes:

[0126] Using F score as the evaluation indicator, the optimal threshold is determined through grid search algorithm and k-fold cross validation;

[0127] The output result of the abnormal log model is binary-classified according to the optimal threshold to obtain the detection result of the abnormal log.

[0128] In a preferred embodiment of the present invention, the anomaly log model outputs the anomaly probability of a data point, which is then binarized using an optimal threshold to ensure clarity. The optimal threshold is determined using the F-score as the evaluation metric, using a grid search algorithm and k-fold cross-validation.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0130] Figure 8 A schematic diagram of the structure of a log inspection device based on machine learning provided in an embodiment of this specification includes:

[0131] The abnormality log acquisition module 301 is used to obtain the logs to be inspected, filter the logs to be inspected, and obtain abnormality logs;

[0132] The template matching module 302 is used to perform abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log;

[0133] A feature extraction module 303 is configured to extract features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0134] Anomaly log detection module 304, configured to input the anomaly log feature set into an anomaly log model based on machine learning to obtain a detection result of the anomaly log;

[0135] The model updating module 305 is configured to obtain an analysis result of the detection result, feed the analysis result back to the abnormality log model, and update the abnormality log model.

[0136] The tracking module 306 is used to create a tracking task and perform tracking inspection on the abnormal log.

[0137] Furthermore, the log inspection device based on machine learning also includes:

[0138] The code scanning module is used to scan the code of historical logs and extract the sample set of historical abnormal logs;

[0139] A sample set parsing module, configured to parse the historical abnormal log sample set using regular expressions to obtain an abnormal log template;

[0140] An importance classification module is used to assign a unique identity to the abnormal log template and classify the abnormal log template into importance levels according to weights;

[0141] A feature processing module, configured to extract a historical abnormal log feature set from the historical abnormal log sample set based on the abnormal log template, and perform feature processing on the historical abnormal log feature set;

[0142] The model generation module is used to input the processed historical abnormal log feature set into the deep neural network model training to obtain the abnormal log model.

[0143] Furthermore, the abnormal log acquisition module 301 includes:

[0144] The log monitoring unit monitors the directory where logs are written in real time, transmits the written logs to the message subscription center in the form of messages, and obtains the logs to be inspected;

[0145] Furthermore, the feature processing module includes:

[0146] a first feature unit, configured to perform statistics on the historical abnormal log sample set according to the unique identity identifier of the abnormal log template, and perform mean and variance processing in combination with the weight of the abnormal log template to obtain statistical features; and / or,

[0147] The second feature unit is configured to obtain a comparison feature by comparing the values ​​before and after the sequence in the historical abnormal log sample set; and / or,

[0148] The third feature unit is used to introduce context features;

[0149] A feature combining unit is configured to combine the statistical features and / or the comparison features and / or the context features to obtain an abnormal log feature set of the historical abnormal log sample set.

[0150] Furthermore, the feature processing module further includes:

[0151] Encoding the discrete features in the historical abnormal log feature set using a single-bit effective code to obtain a binary feature;

[0152] Feature scaling is performed on the historical abnormal log feature set to obtain the processed historical abnormal log feature set.

[0153] Furthermore, the second characteristic unit includes:

[0154] The comparison feature is obtained by comparing the values ​​before and after the sequence in the historical log sample set using a difference and / or ratio comparison method.

[0155] Furthermore, the abnormal log detection module 304 includes:

[0156] Using F score as the evaluation indicator, the optimal threshold is determined through grid search algorithm and k-fold cross validation;

[0157] The output result of the abnormal log model is binary-classified according to the optimal threshold to obtain the detection result of the abnormal log.

[0158] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0159] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0160] S201: Obtain logs to be inspected, filter the logs to be inspected, and obtain abnormal logs;

[0161] S202: performing abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log;

[0162] S203: Extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0163] S204: Inputting the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log;

[0164] S205: Obtain analysis results of the detection results, feed the analysis results back to the abnormality log model, and update the abnormality log model;

[0165] S206: Create a tracking task to track and inspect the abnormal log.

[0166] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0167] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0168] Furthermore, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0169] Furthermore, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0170] S201: Obtain logs to be inspected, filter the logs to be inspected, and obtain abnormal logs;

[0171] S202: performing abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log;

[0172] S203: Extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set;

[0173] S204: Inputting the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log;

[0174] S205: Obtain analysis results of the detection results, feed the analysis results back to the abnormality log model, and update the abnormality log model;

[0175] S206: Create a tracking task to track and inspect the abnormal log.

[0176] Furthermore, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0177] Reference Figure 4 This is a flow chart of a log inspection process based on machine learning provided in an embodiment of this specification.

[0178] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0179] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A log inspection method based on machine learning, characterized in that: include: Obtaining logs to be inspected, screening the logs to be inspected, and obtaining abnormal logs; Performing abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log; wherein the abnormal log template is assigned a unique identity and is divided into importance levels according to weights; Extracting features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set, specifically comprising: performing statistics on a historical abnormal log sample set according to a unique identity identifier of the abnormal log template, and performing mean and variance processing in combination with a weight of the abnormal log template to obtain statistical features; obtaining comparative features by comparing values ​​before and after a sequence in the historical abnormal log sample set; introducing contextual features; and combining the statistical features, the comparative features, and the contextual features to obtain an abnormal log feature set for the historical abnormal log sample set; Inputting the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log; Obtaining an analysis result of the detection result, feeding the analysis result back to the abnormality log model, and updating the abnormality log model; Create a tracking task to track and inspect the abnormal log.

2. The log inspection method based on machine learning according to claim 1, characterized in that: Before obtaining the logs to be inspected, the following steps are performed: Scan historical logs and extract historical abnormal log sample sets; Parsing the historical abnormal log sample set using regular expressions to obtain an abnormal log template; Extracting a historical abnormal log feature set of the historical abnormal log sample set based on the abnormal log template, and performing feature processing on the historical abnormal log feature set; The processed historical abnormal log feature set is input into the deep neural network model training to obtain the abnormal log model.

3. The log inspection method based on machine learning according to claim 2, characterized in that: The step of obtaining the log to be inspected includes: The directory where the logs are written is monitored in real time, and the written logs are transmitted to a message subscription center in the form of messages to obtain the logs to be inspected.

4. The log inspection method based on machine learning according to claim 3, characterized in that: After extracting the historical abnormal log sample set, including: Oversampling is performed on the historical abnormal log sample set.

5. The log inspection method based on machine learning according to claim 4, characterized in that: The feature processing of the historical abnormal log feature set includes: Encoding the discrete features in the historical abnormal log feature set using a single-bit effective code to obtain a binary feature; Feature scaling is performed on the historical abnormal log feature set to obtain the processed historical abnormal log feature set.

6. The log inspection method based on machine learning according to claim 5, characterized in that: The comparison features are obtained by comparing the values ​​before and after the sequence in the historical abnormal log sample set, including: The comparison feature is obtained by comparing the values ​​before and after the sequence in the historical log sample set using a difference and / or ratio comparison method.

7. The log inspection method based on machine learning according to claim 6, characterized in that: The obtaining of the detection result of the abnormal log includes: The F score is used as the evaluation indicator, and the optimal threshold is determined through the grid search algorithm and k-fold cross validation; the output result of the abnormal log model is binary classified according to the optimal threshold to obtain the detection result of the abnormal log.

8. A log inspection device based on machine learning, characterized in that: include: The abnormal log acquisition module is used to obtain the logs to be inspected, filter the logs to be inspected, and obtain abnormal logs; A template matching module is used to perform abnormal log template matching on the abnormal log to obtain an abnormal log template corresponding to the abnormal log; wherein the abnormal log template is assigned a unique identity and is divided into importance levels according to weights; A feature extraction module is configured to extract features from the abnormal log based on the abnormal log template to obtain an abnormal log feature set, specifically comprising: performing statistics on a historical abnormal log sample set according to a unique identity identifier of the abnormal log template, and performing mean and variance processing in combination with the weight of the abnormal log template to obtain statistical features; obtaining comparative features by comparing values ​​before and after a sequence in the historical abnormal log sample set; introducing contextual features; and combining the statistical features, the comparative features, and the contextual features to obtain an abnormal log feature set for the historical abnormal log sample set; An abnormal log detection module is used to input the abnormal log feature set into an abnormal log model based on machine learning to obtain a detection result of the abnormal log; a model updating module, configured to obtain an analysis result of the detection result, feed the analysis result back to the anomaly log model, and update the anomaly log model; The tracking module is used to create a tracking task and track and inspect the abnormal log.

9. An electronic device, wherein: The electronic device includes: A processor; and a memory storing a computer executable program, which, when executed, causes the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images