Log classification method and apparatus, electronic device, and storage medium

Abstract

The embodiment discloses a log classification method and device, electronic equipment and a computer storage medium. The method comprises the following steps: extracting a structured log template for an original log data set; constructing a word vector of the structured log template; searching a network structure of a neural network for log classification by using a gradient descent search method according to the word vector of the structured log template, and obtaining a target network structure; training the neural network under the condition that the network structure of the neural network is the target network structure, and obtaining a trained neural network; and classifying logs based on the trained neural network.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence (AI) technology, and in particular relates to a log classification method, device, electronic device, and storage medium. Background Art

[0002] With the global development of informatization, the number of network devices is growing rapidly, and Internet technology (IT) is playing an increasingly important role in enterprise development. At the same time, IT systems are becoming increasingly complex, and enterprises need to continuously increase the number of servers to maintain the performance of websites and applications. This makes it difficult for operations and maintenance personnel to calmly handle various server issues. During server operation, servers generate a large number of log files, which contain important information about the server's operation. Proper application server log information can help operations and maintenance personnel solve server problems.

[0003] In related technologies, neural networks can be used to classify logs. However, with the increase in servers and businesses, the types of logs are constantly increasing. In this scenario where the types of logs are constantly increasing, the accuracy of using neural networks to classify logs is low. Summary of the Invention

[0004] Embodiments of the present application provide a log classification method, apparatus, electronic device, and computer storage medium.

[0005] This embodiment of the present application provides a log classification method, the method comprising:

[0006] Extract structured log templates from the original log dataset;

[0007] Constructing a word vector for the structured log template;

[0008] According to the word vector of the structured log template, a gradient descent search method is used to search the network structure of the neural network for log classification to obtain a target network structure;

[0009] When the network structure of the neural network is the target network structure, training the neural network to obtain a trained neural network;

[0010] Log classification is performed based on the trained neural network.

[0011] In some embodiments, the neural network includes a first subnetwork for processing time series data and a second subnetwork for text classification; the search method is used to search the network structure of the neural network for log classification based on the word vector of the structured log template using a gradient descent search method to obtain a target network structure, including: searching the network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork based on the word vector of the structured log template using a gradient descent search method to obtain the parameters of the target network structure, the parameters of the target network structure include the searched network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork; and determining the target network structure based on the parameters of the target network structure.

[0012] It can be seen that the embodiment of the present application, by adopting a gradient descent search method, can search for parameter configuration information of the structure of the first subnetwork for processing time series data, the structure of the second subnetwork for text classification, and the connection structure between the first subnetwork and the second subnetwork, thereby more accurately determining the target network structure. Furthermore, when the network structure of the neural network is the target network structure, the accuracy of log classification using the neural network can be improved.

[0013] In some embodiments, the first sub-network is a Bi-directional Long Short-Term Memory (BiLSTM) network. As can be seen, since the BiLSTM network is used to process time series data, and log data has time series characteristics, when the neural network includes a BiLSTM network for processing time series data, the trained neural network can more accurately classify log data.

[0014] In some embodiments, the second sub-network is a Text Convolutional Neural Network (TextCNN). As can be seen, since the TextCNN network is a network for text classification, and log data is text data, when the neural network includes a TextCNN network for text classification, the trained neural network can more accurately classify logs.

[0015] In some embodiments, extracting the structured log template from the original log data set includes: extracting a fixed portion of each log in the original log data set to obtain the structured log template.

[0016] It can be understood that by extracting the fixed parts of each log in the original log dataset, a unified structured log template can be obtained; since the fixed parts of the original log can reflect the basic situation of the original log, the subsequent word vector construction can be carried out more accurately for the unified structured log template.

[0017] In some embodiments, constructing the word vector for the structured log template includes constructing the word vector for the structured log template using a Word2vec model. Because the Word2vec model can not only represent relationships between contexts but also represent vectors of larger word bags, using the Word2vec model can more accurately construct the word vector for the structured log template.

[0018] In some embodiments, the neural network is a deep learning network. Since deep learning networks have strong feature extraction capabilities, logs can be classified more accurately using a trained deep learning network.

[0019] The present application also provides a log classification device, comprising:

[0020] The extraction module is used to extract structured log templates from the original log dataset;

[0021] a processing module configured to construct a word vector for the structured log template; search the network structure of a neural network for log classification using a gradient descent search method based on the word vector of the structured log template to obtain a target network structure; and train the neural network when the network structure of the neural network is the target network structure to obtain a trained neural network;

[0022] A classification module is used to classify logs based on the trained neural network.

[0023] An embodiment of the present application further provides an electronic device, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is configured to run the computer program to execute any one of the above-mentioned log classification methods.

[0024] An embodiment of the present application further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned log classification methods is implemented.

[0025] It can be seen that in the embodiment of the present application, by using the gradient descent search method to search the network structure of the neural network used for log classification, the search for the network structure can be completed efficiently, and a suitable network structure can be searched based on the original log data set. Furthermore, when the network structure of the neural network is the target network structure, the accuracy of log classification using the neural network can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of the log classification method according to an embodiment of the present application;

[0027] Figure 2 This is a schematic diagram of extracting a structured log template in an embodiment of the present application;

[0028] Figure 3 Another flow chart of the log classification method according to an embodiment of the present application;

[0029] Figure 4 This is a structural diagram of a log classification device according to an embodiment of the present application;

[0030] Figure 5 This is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the related art, the following three schemes can be used for log classification: a rule-based log classification scheme, a log classification scheme through machine learning, and a log classification scheme using a deep learning network.

[0032] In a rule-based log classification solution, operations engineers can set different matching rules based on different log formats to easily implement log classification. For example, they can match server log rules and set different rules for similarity classification based on the similarities of various abnormal events.

[0033] Solutions for log classification using machine learning and deep learning networks are emerging with the development of artificial intelligence. For example, solutions for log classification using machine learning can include methods such as clustering and text classification to process log information. Since logs are also a form of text, natural language processing technology can also be incorporated into log processing and log classification solutions. For example, solutions for log classification using deep learning networks can be log classification solutions that utilize natural language processing technology. These solutions that use artificial intelligence technology to process operation and maintenance logs are called Artificial Intelligence for ITOperations (AIOps) solutions. AIOps uses algorithms such as machine learning, deep learning, and reinforcement learning to automatically learn from massive amounts of operation and maintenance data and summarize the inherent characteristics of logs to achieve log classification, which can then be used for decision-making. By executing AIOPs, large amounts of data can be quickly analyzed and processed, effective operation and maintenance decisions can be made, and automated scripts can be executed to achieve overall operation and maintenance of IT systems, ensuring the stability of IT systems.

[0034] Rule-based log classification schemes present the following technical challenges: With the increase in servers, the types of operation and maintenance logs continue to grow, leading to a significant increase in the number of new logs. Relying solely on manual classification and rule matching often requires a large number of staff. Furthermore, manual log classification schemes are prone to misclassification or omission of logs. Currently, businesses must quickly and accurately resolve computer failures. When computer problems arise, relying solely on manual rule-based solutions cannot meet business needs.

[0035] Machine learning-based log classification solutions have the following technical issues: While they can address some of the issues with rule-based log classification solutions, due to the limited capabilities of machine learning for log processing, log classification accuracy is often low. This is due to its limited ability to extract data features, making it incapable of processing a wide variety of logs. This is especially true for logs with less distinct features or fewer specific characteristics. The processing results are often inaccurate, impacting the accuracy of the final classification results.

[0036] Although the solution of using deep learning networks for log classification solves the problem of insufficient feature processing capabilities of machine learning, the wide variety of deep learning networks and different log data sets often require different network structures to process. In addition, with the increase in servers and businesses, a variety of log data sets have emerged, making it difficult to design a network structure that meets actual needs in a timely and accurate manner. Using a network structure that does not meet actual needs for log classification will reduce the accuracy of log classification.

[0037] In response to the above technical problems, the technical solutions of the embodiments of the present application are proposed.

[0038] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely for explaining the embodiments of the present application and are not intended to limit the embodiments of the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions described in the embodiments of the present application can be implemented in any combination.

[0039] It should be noted that, in the embodiments of the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be a portion of a circuit, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus comprising the element.

[0040] An embodiment of the present application provides a log classification method. The log classification method provided in the embodiment of the present application includes a series of steps, but the log classification method provided in the embodiment of the present application is not limited to the recorded steps. Similarly, the log classification device provided in the embodiment of the present application includes a series of modules, but the device provided in the embodiment of the present application is not limited to including the modules explicitly recorded, and can also include modules required to obtain relevant information or perform processing based on the information.

[0041] Figure 1 This is a flow chart of the log classification method according to an embodiment of the present application. Figure 1 As shown, the process may include:

[0042] Step 101: extracting a structured log template from an original log dataset.

[0043] In the embodiment of the present application, the original log data set may include multiple original logs; illustratively, the original log of the original log data set may be the original log of the server.

[0044] Here, the original logs in the original log dataset are usually semi-structured data. In order to effectively process the original logs, it is necessary to preprocess the original logs, that is, to extract structured log templates from the semi-structured logs.

[0045] Step 102: Construct a word vector for the structured log template.

[0046] There are many ways to represent word vectors, and they have different effects on different tasks such as accuracy, classification, and clustering. The embodiment of this application can select an appropriate method according to actual needs to construct the word vector of the structured log template.

[0047] In some embodiments, one-hot encoding, matrix decomposition, a word vector construction scheme based on a neural network language model (NNLM), or a word vector construction scheme based on a recurrent neural network language model (RNNLM) can be used to construct a word vector for a structured log template.

[0048] When sentences are relatively long, using one-hot encoding to construct word vectors will lead to the curse of dimensionality and semantic gaps. When constructing word vectors using matrix factorization, singular value decomposition (SVD) is required, and the solution is very complex. In word vector construction solutions based on NNLM or RNNLM, word vectors are a byproduct, resulting in low word vector construction efficiency.

[0049] In some embodiments of the present application, a Word2vec model may be used to construct a word vector for the structured log template. Exemplarily, the Word2vec model may be a Pos weight.

[0050] Since the Word2vec model can not only represent the relationship between contexts, but also represent the vectors of larger word bags, using the Word2vec model can more accurately construct the word vectors of structured log templates to a certain extent.

[0051] Step 103: Based on the word vector of the structured log template, a gradient descent search method is used to search the network structure of the neural network for log classification to obtain a target network structure.

[0052] Exemplarily, the gradient descent search method can be the Differentiable Architecture Search (DARTS) method; the advantage of the DARTS method is that DARTS is a method based on relaxation and continuation, that is, the DARTS method can relax the structure search problem into a continuous problem, so that it can be optimized by gradient descent. Compared with discrete structure search schemes or schemes that attempt to select network structures based on historical experience, the DARTS method requires a shorter search time, that is, the DARTS method can search for a suitable network structure more quickly.

[0053] DARTS models Neural Architecture Search (NAS) as a bi-level optimization problem, using alternate optimization methods to find the optimal network architecture. DARTS is also a one-shot NAS method, which first builds a supernet and then extracts the optimal subnetwork from it.

[0054] Step 104: When the network structure of the neural network is the target network structure, the neural network is trained to obtain a trained neural network.

[0055] Step 105: Perform log classification based on the trained neural network.

[0056] Here, the logs to be classified can be obtained through online monitoring or other means, and the logs to be classified are classified based on the trained neural network.

[0057] In practical applications, steps 101 to 105 can be implemented based on a processor of an electronic device, and the processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0058] It can be seen that in the embodiment of the present application, by using the gradient descent search method to search the network structure of the neural network used for log classification, the search for the network structure can be completed efficiently, and a suitable network structure can be searched based on the original log data set. Furthermore, when the network structure of the neural network is the target network structure, the accuracy of log classification using the neural network can be improved.

[0059] In the embodiment of the present application, for data sets of different businesses, the data sets of different businesses can be used as original log data sets respectively, and then the network structure is searched according to the technical solution of the embodiment of the present application, and log classification is performed. Therefore, the embodiment of the present application can optimize the network structure for the scenarios of different businesses, make the classification results of log classification more accurate, and can promptly and accurately discover the anomalies existing in the logs. That is, the log classification method of the embodiment of the present application can effectively find the appropriate network architecture based on the characteristics of the data set. The log classification method of the embodiment of the present application can improve the accuracy of log classification and has good portability.

[0060] Furthermore, the embodiments of the present application do not require the setting of manual rules, which can reduce labor costs. In the context of classifying large-scale logs of various types, the embodiments of the present application can improve the efficiency of log classification.

[0061] In some embodiments, the process of extracting the structured log template from the original log data set may include: extracting the fixed and unchanged parts of each log in the original log data set to obtain the structured log template. Figure 2 This is a schematic diagram of extracting a structured log template in an embodiment of the present application. Figure 2 As can be seen in the right side of the structured log template, the number part has been deleted. At the same time, since some prepositions have no meaning, they can also be deleted. For example, "the" and "a" have no actual meaning, so these words will also be deleted.

[0062] Raw logs contain data in two formats: a variable portion (variables) and a fixed portion. Variables can include service start and end times, server IP addresses, and so on. The fixed portion is the primary component of the log. For example, the fixed portion might be the event type and name. This portion reflects the basic details of the raw log and can approximate the log.

[0063] For example, a fault tree (FT-Tree) model may be used to extract the fixed and unchanged parts of each log in the original log data set to obtain a structured log template.

[0064] It can be understood that by extracting the fixed parts of each log in the original log dataset, a unified structured log template can be obtained; since the fixed parts of the original log can reflect the basic situation of the original log, the subsequent word vector construction can be carried out more accurately for the unified structured log template.

[0065] In some embodiments of the present application, the neural network is a deep learning network. Since deep learning networks have strong feature extraction capabilities, logs can be classified more accurately using a trained deep learning network.

[0066] In some embodiments of the present application, the neural network includes a first subnetwork for processing time series data and a second subnetwork for performing text classification.

[0067] Accordingly, based on the word vectors of the structured log template, the gradient descent search method is used to search the network structure of the neural network for log classification, and the target network structure is obtained, including:

[0068] Based on the word vector of the structured log template, a gradient descent search method is used to search the network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork to obtain the parameters of the target network structure. The parameters of the target network structure include the searched network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork;

[0069] Determine the target network structure according to the parameters of the target network structure.

[0070] It can be seen that the embodiment of the present application, by adopting a gradient descent search method, can search for parameter configuration information of the structure of the first subnetwork for processing time series data, the structure of the second subnetwork for text classification, and the connection structure between the first subnetwork and the second subnetwork, thereby more accurately determining the target network structure. Furthermore, when the network structure of the neural network is the target network structure, the accuracy of log classification using the neural network can be improved.

[0071] In some embodiments of the present application, the first sub-network is a BiLSTM network. As can be seen, since the BiLSTM network is used to process time series data, and log data is data with time series characteristics, when the neural network includes a BiLSTM network used to process time series data, the trained neural network can more accurately classify logs.

[0072] In some embodiments of the present application, the second sub-network is a TextCNN network. As can be seen, since the TextCNN network is a network for text classification, and log data is text data, when the neural network includes a TextCNN network for text classification, the trained neural network can more accurately classify logs.

[0073] Figure 3 Another flow chart of the log classification method of the embodiment of the present application is shown in FIG. Figure 3 , the historical log of the training data is the original log sequence, and the original log sequence can be expressed as L, L=[l1,...l n ], where n is an integer greater than 1, and the original log sequence includes n original logs.

[0074] When the training data is the original log data set, the FT-Tree model can be used to analyze (parse) each log in the original log data set, thereby extracting the fixed and unchanging parts of each log in the original log data set to obtain a structured log template; Figure 3 Template1 to Template i-1 Represents the structured log template extracted from the original log dataset, and i is an integer greater than 1.

[0075] After extracting the structured log template, the Word2vec model can be used to construct the word vector of the structured log template; Figure 3 For template Template1 to template Template i-1 The constructed word vectors can be recorded as V1 to V i-1 .

[0076] Based on the word vectors of the structured log template, the DARTS method is used to search the network structure of the neural network used for log classification to obtain the target network structure; after obtaining the target network structure, a neural network can be constructed, which is an adaptive word vector weight AIOps log classification network based on reinforcement learning.

[0077] After obtaining the trained neural network, you can obtain the logs to be classified; refer to Figure 3 The log to be classified is the real-time log obtained through online detection. After extracting the structured log template and constructing the word vector for the real-time log, the word vector corresponding to the real-time log can be input into the trained neural network, so as to classify the real-time log using the trained neural network. For example, referring to Figure 3 Templatei To Template i+1 Represents a part of the structured log template for real-time log extraction, for Template i To Template i+1 The constructed word vectors can be recorded as V i To V i+1 .

[0078] In an exemplary application scenario, the types of log classification may include abnormal logs and normal logs. Therefore, anomaly detection of logs can be achieved through log classification.

[0079] In summary, the embodiment of the present application proposes an adaptive network architecture AIOps log classification method based on DARTS. This method searches for deep learning network architecture through DARTS, and designs an effective network structure based on the corresponding data set, effectively extracts features from operation and maintenance logs, and improves the accuracy of log classification.

[0080] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0081] Based on the log classification method proposed in the above embodiment, the embodiment of the present application further proposes a log classification device.

[0082] Figure 4 This is a structural diagram of the log classification device according to an embodiment of the present application. Figure 4 As shown, the device may include:

[0083] Extraction module 401, used to extract structured log templates from the original log data set;

[0084] Processing module 402 is configured to construct a word vector for the structured log template; based on the word vector of the structured log template, use a gradient descent search method to search the network structure of a neural network for log classification to obtain a target network structure; and if the network structure of the neural network is the target network structure, train the neural network to obtain a trained neural network;

[0085] The classification module 403 is configured to classify logs based on the trained neural network.

[0086] In some embodiments, it is characterized in that the neural network includes a first subnetwork for processing time series data and a second subnetwork for performing text classification;

[0087] The processing module 402 is configured to search the network structure of the neural network for log classification using a gradient descent search method based on the word vector of the structured log template to obtain a target network structure, including:

[0088] Searching, based on the word vector of the structured log template, the network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork using a gradient descent search method to obtain parameters of a target network structure, wherein the parameters of the target network structure include the searched network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork;

[0089] The target network structure is determined according to the parameters of the target network structure.

[0090] In some embodiments, the first sub-network is a BiLSTM network.

[0091] In some embodiments, the second sub-network is a Textcnn network.

[0092] In some embodiments, the extraction module 401 is configured to extract a structured log template from an original log data set, including: extracting a fixed portion of each log in the original log data set to obtain the structured log template.

[0093] In some embodiments, the processing module is used to construct the word vector of the structured log template, including: using a Word2vec model to construct the word vector of the structured log template.

[0094] In some embodiments, the neural network is a deep learning network.

[0095] In practical applications, the extraction module 401 , the processing module 402 and the classification module 403 may all be implemented based on a processor of an electronic device.

[0096] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0097] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0098] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer-executable instructions, and the computer-executable instructions are used to implement any log classification method provided in the embodiment of the present application.

[0099] Accordingly, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored. The computer executable instructions are used to implement any one of the log classification methods provided in the above embodiments.

[0100] The embodiment of the present application also provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 50 may include:

[0101] Memory 51, used for storing executable instructions;

[0102] The processor 52 is configured to implement any one of the above-mentioned log classification methods when executing the executable instructions stored in the memory 51 .

[0103] The processor 52 may be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0104] The above-mentioned computer-readable storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0105] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0106] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0107] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0108] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0109] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0111] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are protected by this application.

Claims

1. A log classification method, characterized in that: The method comprises: Extract structured log templates from the original log dataset; Constructing a word vector for the structured log template; According to the word vector of the structured log template, a gradient descent search method is used to search the network structure of the neural network for log classification to obtain a target network structure; When the network structure of the neural network is the target network structure, training the neural network to obtain a trained neural network; Log classification is performed based on the trained neural network.

2. The method according to claim 1, characterized in that The neural network includes a first subnetwork for processing time series data and a second subnetwork for performing text classification; The method of searching the network structure of the neural network for log classification using a gradient descent search method based on the word vector of the structured log template to obtain a target network structure includes: Searching, based on the word vector of the structured log template, the network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork using a gradient descent search method to obtain parameters of a target network structure, wherein the parameters of the target network structure include the searched network structure parameters of the first subnetwork, the network structure parameters of the second subnetwork, and the connection structure parameters between the first subnetwork and the second subnetwork; The target network structure is determined according to the parameters of the target network structure.

3. The method according to claim 2, characterized in that The first sub-network is a bidirectional long short-term memory BiLSTM network.

4. The method according to claim 2, characterized in that The second sub-network is a Textcnn network.

5. The method according to any one of claims 1 to 4, characterized in that The extracting of the structured log template from the original log data set includes: The fixed and unchanged part of each log in the original log data set is extracted to obtain the structured log template.

6. The method according to any one of claims 1 to 4, characterized in that The constructing of the word vector of the structured log template includes: using a Word2vec model to construct the word vector of the structured log template.

7. The method according to any one of claims 1 to 4, characterized in that The neural network is a deep learning network.

8. A log classification device, characterized in that: The device comprises: The extraction module is used to extract structured log templates from the original log dataset; a processing module configured to construct a word vector for the structured log template; search the network structure of a neural network for log classification using a gradient descent search method based on the word vector of the structured log template to obtain a target network structure; and train the neural network when the network structure of the neural network is the target network structure to obtain a trained neural network; A classification module is used to classify logs based on the trained neural network.

9. An electronic device, characterized in that: comprising a processor and a memory for storing a computer program capable of running on the processor; wherein, The processor is configured to run the computer program to execute the log classification method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log classification method according to any one of claims 1 to 9 is implemented.

Images