Heterogeneous multi-source alarm log classification method and system based on improved TF-IDF and CNN

Abstract

The invention belongs to the technical field of alarm log processing of an electric power data communication network, and discloses a heterogeneous multi-source alarm log classification method and a related system for improving TF-IDF and CNN, and the method achieves the standardized processing of non-standardized and diversified alarm information through the construction of a unified alarm keyword library and the preprocessing of data, and improves the classification efficiency. And the availability and the consistency of the data are enhanced. The improved TF-IDF algorithm adopted by the invention can quantify the importance of keywords in the alarm information, extract high-value features, and improve the accuracy of alarm data feature extraction. The convolutional neural network (CNN) has a strong feature learning capability, can capture a deep pattern from alarm information, improves the recognition capability of a classifier for complex features, and enhances the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of alarm log processing of electric power data communication networks, and in particular relates to a heterogeneous multi-source alarm log classification method and a related system based on improved TF-IDF and CNN. Background Art

[0002] As the scale and complexity of power data communication networks continue to increase, the number of heterogeneous alarm logs generated by devices from different manufacturers continues to grow. These alarm logs contain critical information about network status and device performance, and their accurate analysis and processing are essential to ensure the stable operation and efficiency of the network. However, due to the diversity of these alarm logs in format and structure, they are difficult to be uniformly processed and analyzed.

[0003] In all walks of life, alarm information processing has always been one of the core tasks of network management and operation and maintenance. At present, most studies have achieved good results in the field of alarm information processing, but these alarm information mostly comes from equipment of a single brand or unified standard, and the proposed analysis methods are mostly for processing such information. In a multi-brand network environment, for non-standardized, diverse and complex information, these methods are difficult to directly analyze alarm data and are difficult to effectively deal with the complexity caused by the diversity of equipment. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiency that it is difficult to directly analyze alarm data for non-standardized diverse and complex information, and to provide a heterogeneous multi-source alarm log classification method and a related system that improves TF-IDF and CNN.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a heterogeneous multi-source alarm log classification method based on improved TF-IDF and CNN, comprising the following steps: Obtain alarm logs from routers of different brands and build an alarm keyword library; Pre-process the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm is improved according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; The improved TF-IDF algorithm is used to extract text features of the alarm information data that meet the requirements and obtain text feature vectors; Input the extracted text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; Use the trained CNN classifier to classify the alarm information data that meets the requirements and output the classification results; Based on the classification results, an alarm data classification library is generated.

[0006] A further improvement of the present invention is that the data in the alarm keyword library is preprocessed to obtain the alarm information data that meets the requirements. The specific method is as follows: Duplicate entries in the alarm keyword library are eliminated, and regular expression tools are used to standardize the log format of the data, to further complete the abnormal data with missing key fields, and to further eliminate the abnormal data with incorrect format, so as to obtain alarm information data that meets the requirements.

[0007] A further improvement of the present invention is that the TF-IDF algorithm is improved according to the word position and the inter-class dispersion of the word position. The specific method of obtaining the improved TF-IDF algorithm is as follows: The calculation method of TF-IDF algorithm is as follows:

[0008]

[0009]

[0010] in, is a phrase in the warning information data, For warning information data, is the total number of all phrases in the warning information data, is the warning information dataset, is the number of warning information data containing phrase t, TF-IDF For improved text extraction models; According to the word position information, the TF-IDF algorithm is improved as follows:

[0011] in, is the key feature in the alarm information data. is the weight function of the position of phrase t in the warning information data in document d; According to the inter-class dispersion of word positions, the TF-IDF algorithm is improved as follows:

[0012] in, For the improved TF-IDF algorithm, is the inverse document frequency weighted by the inter-class dispersion,

[0013]

[0014] in, is the total number of occurrences of phrase t in all categories, is the inter-class dispersion.

[0015] A further improvement of the present invention is that the warning information data is classified using a trained CNN classifier, and the specific method of outputting the classification result is as follows: Use convolutional layer processing to extract key features from the warning information data; The pooling layer is used to process the key features after the convolution layer, reducing the dimension of the key features and abstracting the required information; A fully connected layer is used to classify and predict the required amount of information after processing by the pooling layer; The softmax function is used to output the predicted category as the classification result.

[0016] A further improvement of the present invention is that the method further comprises: After outputting the classification results, the classification results are compared with the true values, and the CNN classifier is evaluated based on the comparison results. The evaluation method is as follows:

[0017] in, To evaluate the accuracy, is the number of correct positive class identifications, is the number of correctly identified negative classes, is the number of false positives, is the number of missed positive classes.

[0018] In a second aspect, the present invention provides a heterogeneous multi-source alarm log classification system based on improved TF-IDF and CNN, including: The alarm keyword library construction module is used to obtain the alarm logs of routers of different brands and build an alarm keyword library; The data preprocessing module is used to preprocess the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm improvement module is used to improve the TF-IDF algorithm according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; A text feature vector extraction module is used to extract text features of alarm information data that meet the requirements by using an improved TF-IDF algorithm to obtain a text feature vector; The CNN classifier training module is used to input the extracted text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; The data classification module is used to classify the alarm information data that meets the requirements using the trained CNN classifier and output the classification results; The alarm data classification library generation module is used to generate the alarm data classification library according to the classification results.

[0019] A further improvement of the present invention is that the specific functions of the data preprocessing module are realized by the following method: Duplicate entries in the alarm keyword library are eliminated, and regular expression tools are used to standardize the log format of the data, to further complete the abnormal data with missing key fields, and to further eliminate the abnormal data with incorrect format, so as to obtain alarm information data that meets the requirements.

[0020] A further improvement of the present invention is that the specific functions of the TF-IDF algorithm improvement module are realized by the following method: The calculation method of TF-IDF algorithm is as follows:

[0021]

[0022]

[0023] in, is a phrase in the warning information data, For warning information data, is the total number of all phrases in the warning information data, is the warning information dataset, is the number of warning information data containing phrase t, TF-IDF For improved text extraction models; According to the word position information, the TF-IDF algorithm is improved as follows:

[0024] in, is the key feature in the alarm information data. is the weight function of the position of phrase t in the warning information data in document d; According to the inter-class dispersion of word positions, the TF-IDF algorithm is improved as follows:

[0025] in, For the improved TF-IDF algorithm, is the inverse document frequency weighted by the inter-class dispersion,

[0026]

[0027] in, is the total number of occurrences of phrase t in all categories, is the inter-class dispersion.

[0028] A further improvement of the present invention is that the specific functions of the data classification module are implemented by the following method: Use convolutional layer processing to extract key features from the warning information data; The pooling layer is used to process the key features after the convolution layer, reducing the dimension of the key features and abstracting the required information; A fully connected layer is used to classify and predict the required amount of information after processing by the pooling layer; The softmax function is used to output the predicted category as the classification result.

[0029] A further improvement of the present invention is that it also includes a data verification module, which is used to compare the classification result with the true value after outputting the classification result, and evaluate the CNN classifier based on the comparison result.

[0030] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of improving the heterogeneous multi-source alarm log classification method of TF-IDF and CNN are implemented.

[0031] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of improving the heterogeneous multi-source alarm log classification method of TF-IDF and CNN are implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes the standardized processing of non-standardized and diversified alarm information by constructing a unified alarm keyword library and preprocessing the data, thereby enhancing the availability and consistency of the data. The improved TF-IDF algorithm adopted by the present invention can quantify the importance of keywords in the alarm information, extract high-value features, and improve the accuracy of feature extraction of alarm data. Convolutional Neural Networks (CNN) have powerful feature learning capabilities, can capture deep-level patterns from alarm information, improve the recognition ability of classifiers for complex features, and enhance classification accuracy. The present invention combines the advantages of TF-IDF and CNN, can effectively process the keyword weights of alarm information, and can perform in-depth analysis of unstructured complex information through deep learning models, overcoming the limitations of traditional methods when facing heterogeneous data. The method can be flexibly extended to different application scenarios or new equipment types, automatically generate an alarm database through classification results, realize structured storage of alarm information, and help improve the efficiency of subsequent retrieval, statistical analysis and decision support. In summary, the present invention can significantly improve the processing capability of non-standardized heterogeneous multi-source alarm logs, and has significant advantages in complex information analysis, classification accuracy, and operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention; Figure 2 is a system diagram of the present invention; Figure 3 This is a structural diagram of the improved alarm log classification algorithm; Figure 4 This is a system diagram of Example 5. DETAILED DESCRIPTION

[0034] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0035] Embodiment 1: See also Figure 1 ,The improved TF-IDF and CNN heterogeneous multi-source alarm log classification method includes the following steps: S1, obtain the alarm logs of routers of different brands and build an alarm keyword library.

[0036] S2, pre-processing the data in the alarm keyword library to obtain alarm information data that meets the requirements.

[0037] S3, improving the TF-IDF algorithm according to the word positions and the inter-class dispersion of the word positions to obtain an improved TF-IDF algorithm.

[0038] S4, using the improved TF-IDF algorithm to extract text features of the alarm information data that meets the requirements and obtain a text feature vector.

[0039] S5, inputting the extracted text feature vector into the CNN classifier, performing model training on the CNN classifier, optimizing the parameters of the CNN classifier, and obtaining a trained CNN classifier.

[0040] S6, use the trained CNN classifier to classify the alarm information data that meets the requirements and output the classification results.

[0041] S7, generating an alarm data classification library according to the classification results.

[0042] Embodiment 2: See also Figure 3 , a heterogeneous multi-source alarm log classification system based on TF-IDF algorithm and convolutional neural network, including: The alarm keyword library construction module is used to obtain the alarm logs of routers of different brands and build an alarm keyword library; The data preprocessing module is used to preprocess the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm improvement module is used to improve the TF-IDF algorithm according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; A text feature vector extraction module is used to extract text features of alarm information data that meet the requirements by using an improved TF-IDF algorithm to obtain a text feature vector; The CNN classifier training module is used to input the extracted text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; The data classification module is used to classify the alarm information data that meets the requirements using the trained CNN classifier and output the classification results; The alarm data classification library generation module is used to generate the alarm data classification library according to the classification results.

[0043] The data verification module is used to compare the classification results with the true values ​​after outputting the classification results, and evaluate the CNN classifier based on the comparison results.

[0044] Embodiment 3: The specific steps of improving the heterogeneous multi-source alarm log classification method of TF-IDF and CNN are as follows: Step 1: Collect historical log data of multi-brand routers in different environments such as enterprise networks, data centers, and power data communication networks under different time periods and network conditions in the past two years, filter out all alarm logs from the historical log data, and build an alarm keyword library covering multi-brand router alarm logs based on the alarm logs.

[0045] Step 2: Preprocess the data in the alarm keyword library. First, deduplicate and merge the data to eliminate duplicate alarm log entries and avoid data redundancy. Then, convert all timestamp fields to the international standard ISO 8601 format, and use regular expression tools to uniformly parse the different log formats and terms of routers of different brands. Then, fill in the missing information for the data with missing key fields, and remove the format errors for the information with incorrect format.

[0046] Step three, improve the TF-IDF algorithm according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm.

[0047] The TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is a classic algorithm widely used for text feature extraction. It measures the relative importance of a word by measuring the frequency of a word in a document and its inverse document frequency in the entire corpus. However, the traditional TF-IDF ignores the contextual information and time dependency of words, which is particularly important in alarm logs. Therefore, the present invention proposes an improved TF-IDF algorithm that combines contextual information and time weighting mechanism. Words in alarm logs usually appear in pairs and have specific contextual associations. For example, "Link Down" and "Interface Down" are both signs of device connection failures in the network. If you rely only on a single word for classification, you may not be able to capture these specific event patterns. To this end, the present invention uses bigram extraction technology to further enrich log features.

[0048] Contextual information is crucial to the analysis of alarm logs, because many alarm words appear in combination (for example, "Link Down" and "Interface Down" represent different events). In order to capture this contextual information, the present invention uses bigram extraction technology to expand the traditional TF-IDF calculation.

[0049] Bigram extraction is to extract two adjacent words as a feature and calculate the frequency of the bigram. For example, "Link Down" is extracted as an overall feature instead of calculating the weights of "Link" and "Down" separately.

[0050] The improved TF-IDF algorithm is defined as shown in formulas (1) and (2).

[0051] (1) (2) in, is a phrase in the warning information data, For documentation, is the total number of all phrases in the warning information data, is the warning information dataset, is the number of warning information data containing phrase t, TF-IDF For improved text extraction models.

[0052] In formula (2), IDF measures the prevalence of a word. The more documents a word appears in, the lower its IDF value is, indicating that its discrimination is lower. Compared with the traditional IDF calculation method, the denominator of IDF in the present invention is increased by 1 to avoid the situation where the denominator is 0, and the TF-IDF value is finally defined as formula (3).

[0053] DF(t,D)(3) The higher the TF-IDF value, the higher the importance and discrimination of the word in the current document.

[0054] In the TF-IDF algorithm, the word frequency TF simply counts the number of times a word appears in a document, without considering the position of the word. However, in many cases, the position of a word has a great influence on its importance. For example, the beginning of an alarm log may contain information such as device type and timestamp, while the key alarm information is usually located in the middle or at the end. In order to enhance this information, the present invention introduces position weight. Its definition is shown in formula (4).

[0055] (4) in, is the weight function of the position of the phrase t in the warning information data in the document d, therefore, the present invention can choose to adjust the weight according to the distance of the word from the end of the document. That is, the words closer to the end can be given higher weights because they are more likely to contain key information.

[0056] Inter-Class Dispersion (ICD) means that in a classification task, if a word appears frequently in documents of different categories, its ability to distinguish different categories is low. On the contrary, if a word mainly appears in one category and rarely appears in other categories, such a word is valuable for classification. Traditional TF-IDF only considers the inverse document frequency (IDF) of a word in the entire document collection, without considering the distribution of the word among different categories.

[0057] For the alarm log classification task, some words may appear frequently in a specific category and rarely in other categories. By introducing inter-class distribution, the classification can be made more accurate.

[0058] A common method is to define inter-class dispersion ICD through concepts in information theory, especially using entropy to measure the uniformity of a word's distribution across all categories. Given a word group t, its distribution across K categories can be calculated using formula (5) to calculate its entropy.

[0059] (5) in, is the conditional probability that a given phrase t belongs to category k. This can be estimated using formula (6).

[0060] (6) in, is the number of times phrase t appears in category k, and is the total number of times phrase t appears in all categories. Entropy H(t) represents the uncertainty of the category distribution of phrase t. If a word is evenly distributed in all categories, its entropy value will reach the maximum, indicating that it is not very helpful for classification; on the contrary, if a word mainly appears in one or a few categories, its entropy value is low, indicating that it has a higher classification value.

[0061] Based on the definition of entropy, the inter-class dispersion ICD is shown in formula (7).

[0062] (7) in, is the entropy value when the phrase t is completely evenly distributed in all categories. Theoretically . Where K is the total number of categories. In this way, the inter-class dispersion ICD value ranges from 0 to 1. The closer the inter-class dispersion ICD value is to 1, the more concentrated the category distribution of phrase t is, and the more helpful it is for classification.

[0063] The inverse document frequency combined with the weighted inter-class dispersion is shown in formula (8).

[0064] (8) Combined with the above analysis, the improved TF-IDF algorithm is defined as shown in formula (9).

[0065] (9) Step 4: Divide the preprocessed data into training sets and test sets, input the training set into the CNN classifier for training, and extract key features from warning information from different sources by combining the position information, contextual relationship and distribution of words between categories.

[0066] The present invention uses a convolutional layer to further process the feature vector obtained by the improved TF-IDF. The convolutional layer can capture the local dependencies among these features, such as specific patterns or phrases related to the alarm type. The convolution formula is shown in formula (10).

[0067] (10) in, represents a feature vector of the input. is the weight of the convolution kernel, b is the bias, f It is an activation function, and the present invention selects the Relu activation function.

[0068] A pooling layer is applied after the convolutional layer to reduce the feature dimension and abstract the most informative part. The pooling layer formula is shown in formula (11).

[0069] (11) in, represents a segment of the convolution output, k is the pooling window size.

[0070] Then, the pooled output is passed through a fully connected layer for classification prediction. The present invention selects a softmax function to output the predicted category. The formula of the fully connected layer is shown in formula (12).

[0071] (12) in, and are the weights and biases of the fully connected layer, is the input vector.

[0072] By identifying and learning local patterns (such as the frequent occurrence of specific alarm phrases), the CNN classifier can effectively capture the key information in the alarm log and perform efficient classification.

[0073] Step 5: After the CNN classifier training is completed, the best alarm classification model is saved. The divided test set is sent to the trained alarm classification model, and the classification results and confidence levels of various alarms are obtained through calculation output.

[0074] Step 6: Compare the classification results with the true values ​​and evaluate the model using accuracy and F1 score.

[0075] Accuracy is one of the most common classification indicators. It measures the proportion of all correctly classified samples to the total number of samples. The calculation formula for accuracy is shown in formula (13).

[0076] (13) in, To evaluate the accuracy, TP (True Positives) is the number of correctly identified positive classes, TN (True Negatives) is the number of correctly identified negative classes, FP (False Positives) is the number of false positives, and FN (False Negatives) is the number of false positives.

[0077] The F1 score is the harmonic mean of precision and recall. It is a balanced measure of the two indicators and is particularly suitable for cases with unbalanced class distribution. Its formula can be defined as formula (9).

[0078] (9) The test results of the present invention on some keyword examples of the alarm log are shown in Table 1. As can be seen from Table 1, the F1 score of the present invention is significantly higher than that of the algorithm before the improvement, and the classification task is better completed.

[0079] Table 1 The impact of the improved TF-IDF algorithm on the data

[0080] Embodiment 4: like Figure 3 As shown, the system of this embodiment includes: The data preprocessing module first eliminates duplicate alarm log entries to avoid data redundancy, then converts all timestamp fields into the international standard ISO 8601 format, and uses regular expression tools to unify the different log formats and terms used by different routers. It then completes the missing parts of the alarm data information and removes the wrong format information.

[0081] The alarm information learning module establishes a network that can recognize complex alarm logs of equipment from different manufacturers. The input is multi-source alarm sequence data, and the output is the classification results of various alarms and the corresponding confidence levels. The feature extraction module uses the TF-IDF algorithm that combines the word's position information, contextual relationship, and inter-class distribution information to extract features from the input alarm log sequence; CNN classification module, which uses convolutional neural network to classify the extracted features; The result output module outputs the classification results of the alarm logs, compares the classification results with the true values, and evaluates the model. A structured alarm database is generated to provide support for subsequent data analysis and network optimization.

[0082] The structure of the standardized database of alarm logs finally generated in this embodiment is shown in Table 2.

[0083] Table 2 Example of standardized alarm database structure

[0084] Embodiment 5: See also Figure 4 As shown, the present invention also provides an electronic device 100 for improving the heterogeneous multi-source alarm log classification method of TF-IDF and CNN; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0085] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0086] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0087] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a heterogeneous multi-source alarm log classification method based on an improved TF-IDF and CNN, and the processor 102 may execute the plurality of instructions to implement: Obtain alarm logs from routers of different brands and build an alarm keyword library; Pre-process the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm is improved according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; The improved TF-IDF algorithm is used to extract text features of the alarm information data that meet the requirements and obtain text feature vectors; Input the extracted text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; Use the trained CNN classifier to classify the alarm information data that meets the requirements and output the classification results; Based on the classification results, an alarm data classification library is generated.

[0088] Embodiment 6: If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0089] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the present invention.

Claims

1. Improve the heterogeneous multi-source alarm log classification method based on TF-IDF and CNN, characterized by: The following steps are involved: Obtain alarm logs from routers of different brands and build an alarm keyword library; Pre-process the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm is improved according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; The improved TF-IDF algorithm is used to extract text features of the alarm information data that meet the requirements and obtain text feature vectors; Input the text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; Use the trained CNN classifier to classify the alarm information data that meets the requirements and output the classification results; Based on the classification results, an alarm data classification library is generated.

2. According to the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method of claim 1, it is characterized in that: The specific method for preprocessing the data in the alarm keyword library to obtain alarm information data that meets the requirements is as follows: Duplicate entries in the alarm keyword library are eliminated, and regular expression tools are used to standardize the log format of the data, to further complete the abnormal data with missing key fields, and to further eliminate the abnormal data with incorrect format, so as to obtain alarm information data that meets the requirements.

3. According to the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method of claim 1, it is characterized in that: The TF-IDF algorithm is improved according to the word position and the inter-class dispersion of the word position. The specific method of obtaining the improved TF-IDF algorithm is as follows: The calculation method of TF-IDF algorithm is as follows: in, is a phrase in the warning information data, For documentation, is the total number of all phrases in the warning information data, is the warning information dataset, is the number of warning information data containing phrase t, TF-IDF For improved text extraction models; According to the word position information, the TF-IDF algorithm is improved as follows: in, is the key feature in the alarm information data. The phrase in the warning information data In the documentation The weight function of the position in the According to the inter-class dispersion of word positions, the TF-IDF algorithm is improved as follows: in, For the improved TF-IDF algorithm, is the inverse document frequency weighted by the inter-class dispersion, in, is the total number of occurrences of phrase t in all categories, is the inter-class dispersion.

4. According to the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method of claim 1, it is characterized in that: The specific method of using the trained CNN classifier to classify the alarm information data and output the classification results is as follows: Use convolutional layer processing to extract key features from the warning information data; The pooling layer is used to process the key features after the convolution layer, reducing the dimension of the key features and abstracting the required information; A fully connected layer is used to classify and predict the required amount of information after processing by the pooling layer; The softmax function is used to output the predicted category as the classification result.

5. According to the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method of claim 1, it is characterized in that: The method further includes: After outputting the classification results, the classification results are compared with the true values, and the CNN classifier is evaluated based on the comparison results. The evaluation method is as follows: in, To evaluate the accuracy, is the number of correct positive class identifications, is the number of correctly identified negative classes, is the number of false positives, is the number of missed positive classes.

6. Improve the heterogeneous multi-source alarm log classification system based on TF-IDF and CNN, characterized by: include: The alarm keyword library construction module is used to obtain the alarm logs of routers of different brands and build an alarm keyword library; The data preprocessing module is used to preprocess the data in the alarm keyword library to obtain alarm information data that meets the requirements; The TF-IDF algorithm improvement module is used to improve the TF-IDF algorithm according to the word position and the inter-class dispersion of the word position to obtain an improved TF-IDF algorithm; A text feature vector extraction module is used to extract text features of alarm information data that meet the requirements by using an improved TF-IDF algorithm to obtain a text feature vector; The CNN classifier training module is used to input the extracted text feature vector into the CNN classifier, perform model training on the CNN classifier, optimize the parameters of the CNN classifier, and obtain a trained CNN classifier; The data classification module is used to classify the alarm information data that meets the requirements using the trained CNN classifier and output the classification results; The alarm data classification library generation module is used to generate the alarm data classification library according to the classification results.

7. The improved TF-IDF and CNN heterogeneous multi-source alarm log classification system according to claim 6 is characterized in that: The specific functions of the data preprocessing module are realized through the following methods: Duplicate entries in the alarm keyword library are eliminated, and regular expression tools are used to standardize the log format of the data, to further complete the abnormal data with missing key fields, and to further eliminate the abnormal data with incorrect format, so as to obtain alarm information data that meets the requirements.

8. The improved TF-IDF and CNN heterogeneous multi-source alarm log classification system according to claim 6 is characterized in that: The specific functions of the TF-IDF algorithm improvement module are implemented through the following methods: The calculation method of TF-IDF algorithm is as follows: in, is a phrase in the warning information data, For documentation, is the total number of all phrases in the warning information data, is the warning information dataset, is the number of warning information data containing phrase t, TF-IDF For improved text extraction models; According to the word position information, the TF-IDF algorithm is improved as follows: in, is the key feature in the alarm information data. is the weight function of the position of phrase t in the warning information data in document d; According to the inter-class dispersion of word positions, the TF-IDF algorithm is improved as follows: in, For the improved TF-IDF algorithm, is the inverse document frequency weighted by the inter-class dispersion, in, is the total number of occurrences of phrase t in all categories, is the inter-class dispersion.

9. The improved TF-IDF and CNN heterogeneous multi-source alarm log classification system according to claim 6 is characterized in that: The specific functions of the data classification module are implemented in the following way: Use convolutional layer processing to extract key features from the warning information data; The pooling layer is used to process the key features after the convolution layer, reducing the dimension of the key features and abstracting the required information; A fully connected layer is used to classify and predict the required amount of information after processing by the pooling layer; The softmax function is used to output the predicted category as the classification result.

10. The heterogeneous multi-source alarm log classification system based on improved TF-IDF and CNN according to claim 6, characterized in that: It also includes a data verification module, which is used to compare the classification result with the true value after outputting the classification result, and evaluate the CNN classifier according to the comparison result.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method according to any one of claims 1 to 5 are implemented.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the improved TF-IDF and CNN heterogeneous multi-source alarm log classification method according to any one of claims 1 to 5 are implemented.

Images