Log classification method and system based on convolutional neural network, equipment and medium
A convolutional neural network and classification method technology, applied in the field of storage cluster log management, can solve the problems of large log volume and redundant information, inaccurate positioning, low efficiency, etc., to improve query and access speed, storage space optimization, The effect of improving detection speed and detection accuracy
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0042] Embodiment 1 of the present invention proposes a log classification method based on convolutional neural network, and applies the convolutional neural network model to log classification. The network has been adjusted to improve detection speed and detection accuracy. Improve the efficiency of developers or operators in locating system exceptions, and better maintain the stability and security of the cluster system.
[0043] Log: Network devices, systems, and service programs, etc., will generate an event record called a log during operation; each line of log records the description of related operations such as date, time, user, and action.
[0044] Convolutional Neural Network (CNN): CNN is constructed by imitating the visual mechanism of creatures. It is a type of feedforward neural network that includes convolutional calculations and has a deep structure. It has the ability to learn representations and is also one of the representative algorithms for deep learning. ...
Embodiment 2
[0056] Based on the convolutional neural network-based log classification method in Embodiment 1 of the present invention, a convolutional neural network-based log classification system is also proposed. Such as Figure 5 It is a schematic diagram of a log classification system based on a convolutional neural network proposed in Embodiment 2 of the present invention, the system includes a preprocessing module, a training module and a management module;
[0057] The preprocessing module is used for parsing the obtained original log file into structured data, and performing feature extraction on the structured data to obtain a feature set, and marking the hot and cold samples in the feature set once;
[0058] The training module is used to divide the feature set into a training set and a test set according to a preset ratio, use the hot and cold samples in the training set to train the convolutional neural network, and adjust the convolution kernel and training parameters of the...
Embodiment 3
[0064] The invention also proposes a device comprising:
[0065] memory for storing computer programs;
[0066] When the processor is used to execute the computer program, the method steps are as follows:
[0067] Such as figure 1 It is a flow chart of the log classification method based on the convolutional neural network proposed in Embodiment 1 of the present invention;
[0068] In step S101, the obtained original log file is parsed into structured data, and feature extraction is performed on the structured data to obtain a feature set, and the hot and cold samples in the feature set are marked once;
[0069] Such as figure 2 It is the flow chart of the original system log preprocessing proposed in Embodiment 1 of the present invention; firstly, the original system log is parsed, and the logkey method can be used for parsing, because each log is composed of constants and variables, and the constants are determined by the system For messages printed directly from progra...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More