Data center log classification method and device

Abstract

The present application discloses a data center log classification method and device. The method includes: processing the first log file to be classified in the data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set; calculating the first weight value corresponding to each first valid word by a first preset algorithm according to the first word frequency corresponding to each first valid word, and obtaining a plurality of first weight values; sorting each first valid word according to the plurality of first weight values ​​to obtain a first sequence; obtaining a log parsing model, and matching the first sequence by a log template tree in the log parsing model to obtain a classification result of the first log file. The present application solves the technical problem in the related art that the fault log is parsed and analyzed according to the word frequency, resulting in relatively low processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method and device for classifying logs of a data center. Background Art

[0002] For large data center networks, device logs are one of the important ways to obtain network status, but the huge amount of logs and the various formats of logs pose great challenges to the extraction of effective information. Mainstream log analysis methods include log storage, log parsing, log mining and other steps. Operation and maintenance personnel of large-scale data centers can analyze massive logs through efficient methods to obtain rapid and accurate abnormal alarms, fault diagnosis, fault repair decision-making and other capabilities. Log parsing is an important part of the log analysis process. Its principle is to identify the variables and unchanged parts (templates) in the original logs, so as to represent the original logs with log templates, which greatly reduces redundant information. Through log parsing, the original log data will be converged into log templates or log events, which provides a basis for subsequent further analysis; the public traditional log parsing algorithms are mainly based on frequent item mining, word frequency trees, clustering algorithms, etc. The shortcomings of the existing technology mainly exist in the following aspects:

[0003] 1. Based on laboratory data sets, the data are relatively regular and no unified data cleaning technology has been summarized.

[0004] 2. In an actual industrial environment, constructing a template based solely on word frequency will cause the model to expand rapidly, making it impossible to operate and maintain subsequent links.

[0005] 3. Due to changes, upgrades, holidays and other factors, the word frequency of industrial log data changes dramatically. The existing technology updates and parses through word frequency. The dramatic changes in word frequency will lead to problems such as too many invalid templates.

[0006] 4. In industrial logs, there are abnormally high-frequency variables (such as time and date, key customer UID, etc.), which cannot be identified by simply relying on word frequency.

[0007] 5. The existing technical solutions have problems such as slow log parsing template generation and log parsing speed, and cannot adapt to industrial data.

[0008] With respect to the problem that the fault logs are parsed and analyzed according to word frequency in the above-mentioned related technologies, resulting in relatively low processing efficiency, no effective solution has been proposed yet. Summary of the invention

[0009] The embodiments of the present application provide a data center log classification method and device to at least solve the technical problem in the related art of parsing and analyzing fault logs according to word frequency, resulting in relatively low processing efficiency.

[0010] According to one aspect of an embodiment of the present application, a method for classifying logs of a data center is provided, comprising: processing a first log file to be classified of the data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule; calculating a first weight value corresponding to each first valid word by a first preset algorithm based on the first word frequency corresponding to each first valid word, and obtaining a plurality of first weight values; sorting each first valid word based on the plurality of first weight values, and obtaining a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; obtaining a log parsing model, and matching the first sequence by a log template tree in the log parsing model, and obtaining a classification result of the first log file.

[0011] According to one aspect of an embodiment of the present application, a log classification method is provided, comprising: processing a first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule; based on the first word frequency corresponding to each first valid word, calculating a first weight value corresponding to each first valid word by a first preset algorithm to obtain a plurality of first weight values; sorting each first valid word according to the plurality of first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; obtaining a log parsing model, and matching the first sequence by a log template tree in the log parsing model to obtain a classification result of the first log file.

[0012] Furthermore, processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set includes: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain the multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

[0013] Furthermore, performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set includes: identifying and screening the multiple initial valid words through the second preset algorithm to obtain the first valid word set; performing noise reduction processing on the word frequency of each first valid word in the first valid word set through the second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0014] Furthermore, the first preset algorithm is in the form of: Wherein, is the first weight value of the first valid word, is the word frequency of each first valid word when the first log file is classified last time, is the first word frequency, is a constant between 0 and 1, c is a constant between 0 and 7, which is the number of times the valid word i does not appear consecutively.

[0015] Furthermore, sorting each first valid word according to the multiple first weight values ​​to obtain a first sequence includes: filtering out first weight values ​​greater than a first preset value from the multiple first weight values ​​to obtain multiple filtered first weight values; sorting each first valid word according to the multiple filtered first weight values ​​to obtain the first sequence.

[0016] Furthermore, after obtaining a log parsing model and matching the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file, the method further includes: iteratively updating the log parsing model according to the classification result of the first log file.

[0017] Furthermore, obtaining a log parsing model includes: obtaining a plurality of sample log files, and processing the sample log files to obtain a plurality of second valid word sets and a second word frequency corresponding to each second valid word in the second valid word set, wherein the second valid word is a word remaining after the sample log file is processed according to the first preset rule; according to the second word frequency corresponding to each second valid word, calculating the second weight value corresponding to each second valid word by the first preset algorithm to obtain a plurality of second weight values; sorting the second valid words in each second valid word set according to the plurality of second weight values ​​to obtain a plurality of second sequences, wherein the second sequence is composed of the second valid words of the sample log file and the second weight values ​​of the second valid words; constructing a template tree corresponding to the second sequence according to each second sequence to obtain a plurality of first log template trees, and saving the plurality of first log template trees to a template library; assembling the plurality of log template trees in the template library to obtain a log template forest, and using the log template forest as the log parsing model.

[0018] Furthermore, constructing a template tree corresponding to each second sequence according to the second sequence to obtain multiple first log template trees includes: determining whether there is a second log template tree with the same root node as the second sequence in the template library; if the second log template tree does not exist, constructing the first log template tree according to the second sequence.

[0019] Furthermore, the method further includes: if the second log template tree exists, inserting the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree.

[0020] Furthermore, before assembling the multiple log template trees in the template library to obtain the log template forest, the method also includes: traversing each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; pruning the valid words in the multiple third sequences according to a second preset rule to obtain a processed third sequence; sorting the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculating the similarity of the multiple fourth sequences by a third preset algorithm to obtain multiple similarity coefficients; screening out similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merging the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and saving the merged log template tree to the template library.

[0021] According to another aspect of an embodiment of the present application, a method for classifying logs of a data center is also provided, including: receiving a first log file to be classified of a data center sent by a client; processing the first log file in a cloud server to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to a first preset rule, and according to the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain multiple first weight values; sorting each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word, and matching the first sequence by a log template tree in a log parsing model to obtain a classification result of the first log file; and returning the classification result to the client.

[0022] According to another aspect of an embodiment of the present application, a data center log classification device is also provided, including: a first processing unit, used to process a first log file to be classified in the data center, and obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to a first preset rule; a first calculation unit, used to calculate a first weight value corresponding to each first valid word by a first preset algorithm based on the first word frequency corresponding to each first valid word, and obtain multiple first weight values; a second processing unit, used to sort each first valid word according to the multiple first weight values, and obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; a matching unit, used to obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0023] Furthermore, the first processing unit includes: a cleaning module, used to perform data cleaning on the first log file through a preset regular expression to obtain a processed first log file; a statistical module, used to perform initial valid word statistics on the processed first log file to obtain the multiple initial valid words; a first processing module, used to perform noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

[0024] Furthermore, the first processing module includes: a screening submodule, used to identify and screen the multiple initial valid words through the second preset algorithm to obtain the first valid word set; a processing submodule, used to perform noise reduction processing on the word frequency of each first valid word in the first valid word set through the second preset algorithm to obtain the first word frequency corresponding to each first valid word.

[0025] Furthermore, the first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0026] Furthermore, the second processing unit includes: a screening module, used to screen out first weight values ​​greater than a first preset value from the multiple first weight values ​​to obtain the screened multiple first weight values; a first sorting module, used to sort each first valid word according to the screened multiple first weight values ​​to obtain the first sequence.

[0027] Furthermore, the device also includes: an updating unit, which is used to iteratively update the log parsing model according to the classification result of the first log file after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file.

[0028] Further, the matching unit includes: a second processing module, which is used to obtain multiple sample log files, and process the sample log files to obtain multiple second effective word sets and second word frequencies corresponding to each second effective word in the second effective word sets, wherein the second effective words are the words remaining after the sample log files are processed according to the first preset rule; a calculation module, which is used to calculate the second weight value corresponding to each second effective word by the first preset algorithm according to the second word frequency corresponding to each second effective word, and obtain multiple second weight values; a second sorting module, which is used to sort the second effective words in each second effective word set according to the multiple second weight values, and obtain multiple second sequences, wherein the second sequence consists of the second effective words of the sample log files and the second weight values ​​of the second effective words; a construction module, which is used to construct a template tree corresponding to the second sequence according to each second sequence, obtain multiple first log template trees, and save the multiple first log template trees to a template library; a building module, which is used to build multiple log template trees in the template library to obtain a log template forest, and use the log template forest as the log parsing model.

[0029] Furthermore, the construction module includes: a judgment submodule, used to judge whether there is a second log template tree with the same root node as the second sequence in the template library; and a construction submodule, used to construct the first log template tree according to the second sequence if the second log template tree does not exist.

[0030] Furthermore, the device further comprises: an inserting module, configured to insert the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree if the second log template tree exists.

[0031] Furthermore, the device also includes: a traversal unit, which is used to traverse each log template tree in the template library before assembling the multiple log template trees in the template library to obtain a log template forest, so as to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; a pruning unit, which is used to prune the valid words in the multiple third sequences according to a second preset rule to obtain a processed third sequence; a sorting unit, which is used to sort the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; a second calculation unit, which is used to calculate the similarity of the multiple fourth sequences by a third preset algorithm to obtain multiple similarity coefficients; a screening unit, which is used to screen out similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; a third processing unit, which is used to merge the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and save the merged log template tree to the template library.

[0032] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the log classification methods described above.

[0033] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the program executes any one of the log classification methods described above when running.

[0034] In the embodiment of the present application, the first log file to be classified in the data center is processed to obtain the first valid word set and the first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; according to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by the first preset algorithm to obtain multiple first weight values; according to the multiple first weight values, each first valid word is sorted and processed to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; the log parsing model is obtained, and the first sequence is matched by the log template tree in the log parsing model to obtain the classification result of the first log file, which solves the technical problem of low processing efficiency caused by parsing and analyzing fault logs according to word frequency in the related technology. The first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, thereby achieving the effect of improving log processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present invention;

[0037] Figure 2 It is a flow chart of a data center log classification method provided in accordance with the first embodiment of the present application;

[0038] Figure 3 This is a flow chart of obtaining a log parsing model provided in Example 1 of the present application;

[0039] Figure 4 is a schematic diagram of a log template tree provided according to Embodiment 1 of the present application;

[0040] Figure 5 It is a schematic diagram of constructing a log template tree according to the first embodiment of the present application;

[0041] Figure 6 is a flow chart of a log classification method provided according to Embodiment 2 of the present application;

[0042] Figure 7 is a flow chart of a log classification method provided according to Embodiment 3 of the present application;

[0043] Figure 8 is a schematic diagram of a log classification device provided according to Embodiment 4 of the present application;

[0044] Fig. 9 It is a schematic diagram of a computer terminal provided according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] Example 1

[0048] According to an embodiment of the present application, a method for classifying logs of a data center is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a data center log classification method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0050] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0051] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the classification method of the logs of the data center in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the classification method of the logs of the data center mentioned above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0054] Under the above operating environment, this application provides Figure 2 The following table shows the classification method of logs in the data center. Figure 2 This is a flowchart of a method for classifying logs of a data center according to the first embodiment of the present application.

[0055] Step S201, processing the first log file to be classified in the data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule.

[0056] Specifically, the first log file to be classified is processed, redundant information is removed, and a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set are obtained. For example, the first valid word set after processing a certain log is {DIAG (valid word): 25 (word frequency), DEVIP: 55, STA: 22, DELETED: 8, DETECTION: 5, SESS: 10}, wherein the valid word is data information after redundant information (for example, variables / special symbols, etc.) is removed.

[0057] Step S202 , according to the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain a plurality of first weight values.

[0058] Specifically, the first weight value of each first valid word is calculated according to the first word frequency through the first preset algorithm. For example, the weight calculation is performed for the first valid word set, and the result is: {DIAG (valid word): 80 (weight), DEVIP: 100, STA: 75, DELETED: 30, DETECTION: 20, SESS: 60}.

[0059] Step S203, sorting each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word.

[0060] Specifically, the first weight value is obtained through the above method, and the first valid words are sorted to obtain a first sequence. For example, the first sequence is: {DEVIP (word): 100 (weight), DIAG: 80, STA: 75, SESS: 60, DELETED: 30, DETECTION: 20}.

[0061] Step S204: acquiring a log parsing model, and matching the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0062] Specifically, the first sequence is input into the log parsing model, matched with the log template tree in the log parsing model, and the classification result of the first log file is obtained.

[0063] In summary, the first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, which not only improves the processing efficiency, but also improves the accuracy of log classification.

[0064] Optionally, in the data center log classification method provided in Example 1 of the present invention, the first log file to be classified of the data center is processed to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain the first valid word set and the first word frequency corresponding to each first valid word in the first valid word set.

[0065] Specifically, a regular expression is used to remove the variables / special symbols that meet the conditions in the first log file, and then the initial valid words of the processed first log file are counted to obtain multiple initial valid words. In order to avoid the initial valid words also including phrases that do not meet the requirements, the initial valid words are processed by a word frequency noise reduction algorithm (i.e., the second preset algorithm mentioned above) to obtain a first valid word set and a first word frequency of the first valid word. Through the above steps, data cleaning of the first log file can effectively remove redundant and invalid information and improve subsequent matching efficiency.

[0066] Optionally, in the data center log classification method provided in Example 1 of the present invention, multiple initial valid words are denoised by a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: identifying and screening multiple initial valid words by a second preset algorithm to obtain a first valid word set; and denoising the word frequency of each first valid word in the first valid word set by a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0067] Specifically, the word frequency denoising algorithm identifies and removes the noise words (high word frequency variables) of the initial valid words to obtain the first valid word, and then processes the word frequency of the first valid word to smooth the fluctuation range of the word frequency to obtain the first word frequency. Through the above-mentioned word frequency denoising algorithm, the word frequency is smoothed to solve the problem of drastic changes in the word frequency of industrial-level logs; and by removing high word frequency variables, the accuracy of log classification is greatly improved. It should be noted that the noise words are not always unchanged. As the classification of logs proceeds, the noise words will be continuously improved and updated.

[0068] Optionally, in the data center log classification method provided in the first embodiment of the present invention, the first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0069] Specifically, the first preset algorithm is in the form of: The above algorithm reduces the impact of word frequency fluctuations on classification results. For the first time to classify the log file, its p i The value is 0. Classification processing is not performed only once for each log file. Through repeated classification processing, the accuracy of the weight value can be further improved, thereby improving the accuracy of log classification.

[0070] Optionally, in the data center log classification method provided in the first embodiment of the present invention, each first valid word is sorted according to multiple first weight values ​​to obtain a first sequence, including: filtering out a first weight value greater than a first preset value from multiple first weight values ​​to obtain multiple filtered first weight values; and sorting each first valid word according to the filtered multiple first weight values ​​to obtain a first sequence.

[0071] Specifically, all the first weight values ​​are screened again. The threshold value (i.e., the first preset value) is used for filtering, and the valid words whose weight value is not greater than the threshold value are not sorted. Through this step, the valid words are further screened, thereby improving the matching efficiency.

[0072] Optionally, in the data center log classification method provided in Example 1 of the present invention, after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method also includes: iteratively updating the log parsing model based on the classification result of the first log file.

[0073] Specifically, the log parsing model is iteratively updated according to the classification result of the first log file to improve the accuracy of the log parsing model.

[0074] Optionally, in the data center log classification method provided in the first embodiment of the present invention, obtaining the log parsing model includes: obtaining multiple sample log files, and processing the sample log files to obtain multiple second valid word sets and second word frequencies corresponding to each second valid word in the second valid word sets, wherein the second valid words are the words remaining after the sample log files are processed according to the first preset rule; based on the second word frequency corresponding to each second valid word, calculating the second weight value corresponding to each second valid word by a first preset algorithm to obtain multiple second weight values; sorting the second valid words in each second valid word set according to the multiple second weight values ​​to obtain multiple second sequences, wherein the second sequence is composed of the second valid words of the sample log file and the second weight values ​​of the second valid words; constructing a template tree corresponding to the second sequence according to each second sequence to obtain multiple first log template trees, and saving the multiple first log template trees to the template library; assembling the multiple log template trees in the template library to obtain a log template forest, and using the log template forest as the log parsing model.

[0075] Specifically, the flowchart of obtaining the log parsing model is as follows: Figure 3 As shown, the sample log file is preprocessed, mainly for data cleaning. During the cleaning process, variables, symbols, etc. are cleared, and the current word can also be filtered through a blacklist algorithm. Then, the valid words in the sample log file are counted, and the word frequency is processed using a word frequency noise reduction algorithm to obtain a plurality of second valid word sets and a second word frequency corresponding to each second valid word in the second valid word set; then, the second word frequency is used to calculate the second weight value of the second valid word using a weight algorithm (i.e., the first preset algorithm mentioned above), and the second valid words are sorted according to the second weight value to obtain a second sequence; a log tree is constructed using the second sequence to obtain a plurality of first log template trees, for example, Figure 4 The schematic diagram of the log template tree is shown in Figure 1. Finally, all the log template trees form the final log template forest, which is the log parsing model. Through the above steps, the log parsing model obtained greatly improves the processing speed and ability of log classification, and can concurrently process billions of industrial logs in a short time.

[0076] Optionally, in the data center log classification method provided in Example 1 of the present invention, a template tree corresponding to the second sequence is constructed based on each second sequence to obtain multiple first log template trees, including: determining whether there is a second log template tree in the template library that is identical to the root node of the second sequence; if the second log template tree does not exist, constructing the first log template tree based on the second sequence.

[0077] Specifically, when constructing the first log template tree, it is first necessary to determine whether the current template library has a second log template tree with the same root node as the second sequence. If not, the first log template tree from the root node to the leaf node is formed through the second sequence. Through the above steps, invalid redundant data can be avoided and the subsequent matching efficiency with log files can be improved.

[0078] Optionally, in the data center log classification method provided in Embodiment 1 of the present invention, the method further includes: if there is a second log template tree, inserting the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree.

[0079] Specifically, when there is a second log template tree, the following two sequences are used as examples for explanation: Sequence 1: {DEVIP (word): 100 (weight), DIAG: 80, STA: 75, SESS: 60, DELETED: 30, DETECTION: 20}; Sequence 2: {DEVIP: 100, CONTROL: 90, DIAG: 80, EXPIRED, TIME: 30, DETECTION: 20}. The processing process is as follows Figure 5 As shown, the valid words of sequence 2 are inserted into the log template tree corresponding to sequence 1 in order. Through the above steps, for log template trees with the same root node, only one is retained, which can effectively reduce the storage pressure of the log parsing model.

[0080] Optionally, in the data center log classification method provided in the first embodiment of the present invention, before the multiple log template trees in the template library are assembled to obtain the log template forest, the method also includes: traversing each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; pruning the valid words in the multiple third sequences according to the second preset rule to obtain the processed third sequences; sorting the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculating the similarity of the multiple fourth sequences by the third preset algorithm to obtain multiple similarity coefficients; screening out the similarity coefficient greater than the second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merging the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and saving the merged log template tree to the template library.

[0081] Specifically, since the log parsing model will be updated regularly after it is formed, the weight of the updated valid words will change, and then the log template tree will also change, so it is necessary to merge and eliminate invalid or redundant log template trees. The merging process is: traverse from the root node of each log template tree to the leaf node, obtain the third sequence of the tree, delete the valid words with a weight value less than the threshold in the sequence (that is, the second preset rule mentioned above), and obtain the processed third sequence. Selectively deleting valid words can effectively improve the calculation efficiency of the similarity coefficient. Sort the valid words in the processed third sequence according to the first letter to obtain a new sequence, then perform similarity calculation to obtain multiple similarity coefficients, and finally merge the log template trees according to the similarity coefficient. If the similarity coefficient is higher than the second preset value (for example, 90%), then the log template trees corresponding to the similarity coefficient are merged. For example, if there are two log template sequences {ABDFE}-sort-->{ABCDEF}; {BDEFA}-sort-->{ABCDEF}, it can be seen that the two sequences are consistent and can be merged with each other.

[0082] The elimination of log template trees is based on the life cycle control of log template trees. If a log template tree fails to match new logs for a long time, the log template tree will be eliminated, which further reduces the number of log template trees in the log template forest. By merging and eliminating log template trees, the log parsing model can be effectively converged and the matching efficiency can be improved.

[0083] In summary, in the data center log classification method provided by the first embodiment of the present invention, by processing the first log file to be classified in the data center, a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set are obtained, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; according to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by the first preset algorithm to obtain multiple first weight values; according to the multiple first weight values, each first valid word is sorted and processed to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; a log parsing model is obtained, and the first sequence is matched through the log template tree in the log parsing model to obtain the classification result of the first log file, which solves the technical problem that the fault log is parsed and analyzed according to the word frequency in the related art, resulting in relatively low processing efficiency. The first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, thereby achieving the effect of improving the log processing efficiency.

[0084] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0086] Example 2

[0087] This application provides Figure 6 The log classification method shown. Figure 6 It is a flowchart of the log classification method according to the second embodiment of the present application.

[0088] Step S601 : Process the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule.

[0089] Specifically, the first log file to be classified is processed, redundant information is removed, and a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set are obtained. For example, the first valid word set after processing a certain log is {DIAG (valid word): 25 (word frequency), DEVIP: 55, STA: 22, DELETED: 8, DETECTION: 5, SESS: 10}, wherein the valid word is data information after redundant information (for example, variables / special symbols, etc.) is removed.

[0090] Step S602 , according to the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain a plurality of first weight values.

[0091] Specifically, the first weight value of each first valid word is calculated according to the first word frequency through the first preset algorithm. For example, the weight calculation is performed for the first valid word set, and the result is: {DIAG (valid word): 80 (weight), DEVIP: 100, STA: 75, DELETED: 30, DETECTION: 20, SESS: 60}.

[0092] Step S603, sorting each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word.

[0093] Specifically, the first weight value is obtained through the above method, and the first valid words are sorted to obtain a first sequence. For example, the first sequence is: {DEVIP (word): 100 (weight), DIAG: 80, STA: 75, SESS: 60, DELETED: 30, DETECTION: 20}.

[0094] Step S604: obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0095] Specifically, the first sequence is input into the log parsing model, matched with the log template tree in the log parsing model, and the classification result of the first log file is obtained.

[0096] In summary, the first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, which not only improves the processing efficiency, but also improves the accuracy of log classification.

[0097] Optionally, in the log classification method provided in Embodiment 2 of the present invention, the first log file to be classified is processed to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain the first valid word set and the first word frequency corresponding to each first valid word in the first valid word set.

[0098] Specifically, a regular expression is used to remove the variables / special symbols that meet the conditions in the first log file, and then the initial valid words of the processed first log file are counted to obtain multiple initial valid words. In order to avoid the initial valid words also including phrases that do not meet the requirements, the initial valid words are processed by a word frequency noise reduction algorithm (i.e., the second preset algorithm mentioned above) to obtain a first valid word set and a first word frequency of the first valid word. Through the above steps, data cleaning of the first log file can effectively remove redundant and invalid information and improve subsequent matching efficiency.

[0099] Optionally, in the log classification method provided in Embodiment 2 of the present invention, noise reduction processing is performed on multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: identifying and screening multiple initial valid words through a second preset algorithm to obtain a first valid word set; noise reduction processing is performed on the word frequency of each first valid word in the first valid word set through a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0100] Specifically, the word frequency denoising algorithm identifies and removes the noise words (high word frequency variables) of the initial valid words to obtain the first valid word, and then processes the word frequency of the first valid word to smooth the fluctuation range of the word frequency to obtain the first word frequency. Through the above-mentioned word frequency denoising algorithm, the word frequency is smoothed to solve the problem of drastic changes in the word frequency of industrial-level logs; and by removing high word frequency variables, the accuracy of log classification is greatly improved. It should be noted that the noise words are not always unchanged. As the classification of logs proceeds, the noise words will be continuously improved and updated.

[0101] Optionally, in the log classification method provided in the second embodiment of the present invention, the first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0102] Specifically, the first preset algorithm is in the form of: The above algorithm reduces the impact of word frequency fluctuations on classification results. For the first time to classify the log file, its p iThe value is 0. Classification processing is not performed only once for each log file. Through repeated classification processing, the accuracy of the weight value can be further improved, thereby improving the accuracy of log classification.

[0103] Optionally, in the log classification method provided in Embodiment 2 of the present invention, each first valid word is sorted according to multiple first weight values ​​to obtain a first sequence, including: filtering out a first weight value greater than a first preset value from multiple first weight values ​​to obtain multiple filtered first weight values; and sorting each first valid word according to the multiple filtered first weight values ​​to obtain a first sequence.

[0104] Specifically, all the first weight values ​​are screened again. The threshold value (i.e., the first preset value) is used for filtering, and the valid words whose weight value is not greater than the threshold value are not sorted. Through this step, the valid words are further screened, thereby improving the matching efficiency.

[0105] Optionally, in the log classification method provided in Example 2 of the present invention, after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method also includes: iteratively updating the log parsing model based on the classification result of the first log file.

[0106] Specifically, the log parsing model is iteratively updated according to the classification result of the first log file to improve the accuracy of the log parsing model.

[0107] Optionally, in the log classification method provided in the second embodiment of the present invention, obtaining the log parsing model includes: obtaining multiple sample log files, and processing the sample log files to obtain multiple second valid word sets and second word frequencies corresponding to each second valid word in the second valid word sets, wherein the second valid words are the words remaining after the sample log files are processed according to the first preset rule; according to the second word frequency corresponding to each second valid word, calculating the second weight value corresponding to each second valid word by a first preset algorithm to obtain multiple second weight values; sorting the second valid words in each second valid word set according to the multiple second weight values ​​to obtain multiple second sequences, wherein the second sequence is composed of the second valid words of the sample log file and the second weight values ​​of the second valid words; constructing a template tree corresponding to the second sequence according to each second sequence to obtain multiple first log template trees, and saving the multiple first log template trees to the template library; assembling the multiple log template trees in the template library to obtain a log template forest, and using the log template forest as the log parsing model.

[0108] Specifically, the flowchart of obtaining the log parsing model is as follows: Figure 3As shown, the sample log file is preprocessed, mainly for data cleaning. During the cleaning process, variables, symbols, etc. are cleared, and the current word can also be filtered through a blacklist algorithm. Then, the valid words in the sample log file are counted, and the word frequency is processed using a word frequency noise reduction algorithm to obtain a plurality of second valid word sets and a second word frequency corresponding to each second valid word in the second valid word set; then, the second word frequency is used to calculate the second weight value of the second valid word using a weight algorithm (i.e., the first preset algorithm mentioned above), and the second valid words are sorted according to the second weight value to obtain a second sequence; a log tree is constructed using the second sequence to obtain a plurality of first log template trees, for example, Figure 4 The schematic diagram of the log template tree is shown in Figure 1. Finally, all the log template trees form the final log template forest, which is the log parsing model. Through the above steps, the log parsing model obtained greatly improves the processing speed and ability of log classification, and can concurrently process billions of industrial logs in a short time.

[0109] Optionally, in the log classification method provided in the second embodiment of the present invention, a template tree corresponding to the second sequence is constructed based on each second sequence to obtain multiple first log template trees, including: determining whether there is a second log template tree with the same root node as the second sequence in the template library; if the second log template tree does not exist, constructing the first log template tree based on the second sequence.

[0110] Specifically, when constructing the first log template tree, it is first necessary to determine whether the current template library has a second log template tree with the same root node as the second sequence. If not, the first log template tree from the root node to the leaf node is formed through the second sequence. Through the above steps, invalid redundant data can be avoided and the subsequent matching efficiency with log files can be improved.

[0111] Optionally, in the log classification method provided in the second embodiment of the present invention, the method further includes: if there is a second log template tree, inserting the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree.

[0112] Specifically, when there is a second log template tree, the following two sequences are used as examples for explanation: Sequence 1: {DEVIP (word): 100 (weight), DIAG: 80, STA: 75, SESS: 60, DELETED: 30, DETECTION: 20}; Sequence 2: {DEVIP: 100, CONTROL: 90, DIAG: 80, EXPIRED, TIME: 30, DETECTION: 20}. The processing process is as follows Figure 5As shown, the valid words of sequence 2 are inserted into the log template tree corresponding to sequence 1 in order. Through the above steps, for log template trees with the same root node, only one is retained, which can effectively reduce the storage pressure of the log parsing model.

[0113] Optionally, in the log classification method provided in the second embodiment of the present invention, before the multiple log template trees in the template library are assembled to obtain the log template forest, the method also includes: traversing each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; pruning the valid words in the multiple third sequences according to the second preset rule to obtain processed third sequences; sorting the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculating the similarity of the multiple fourth sequences by a third preset algorithm to obtain multiple similarity coefficients; screening out similarity coefficients greater than the second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merging the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and saving the merged log template tree to the template library.

[0114] Specifically, since the log parsing model will be updated regularly after it is formed, the weight of the updated valid words will change, and then the log template tree will also change, so it is necessary to merge and eliminate invalid or redundant log template trees. The merging process is: traverse from the root node of each log template tree to the leaf node, obtain the third sequence of the tree, delete the valid words with a weight value less than the threshold in the sequence (that is, the second preset rule mentioned above), and obtain the processed third sequence. Selectively deleting valid words can effectively improve the calculation efficiency of the similarity coefficient. Sort the valid words in the processed third sequence according to the first letter to obtain a new sequence, then perform similarity calculation to obtain multiple similarity coefficients, and finally merge the log template trees according to the similarity coefficient. If the similarity coefficient is higher than the second preset value (for example, 90%), then the log template trees corresponding to the similarity coefficient are merged. For example, if there are two log template sequences {ABDFE}-sort-->{ABCDEF}; {BDEFA}-sort-->{ABCDEF}, it can be seen that the two sequences are consistent and can be merged with each other.

[0115] The elimination of log template trees is based on the life cycle control of log template trees. If a log template tree fails to match new logs for a long time, the log template tree will be eliminated, which further reduces the number of log template trees in the log template forest. By merging and eliminating log template trees, the log parsing model can be effectively converged and the matching efficiency can be improved.

[0116] In summary, in the log classification method provided by the second embodiment of the present invention, by processing the first log file to be classified, a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set are obtained, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; according to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by the first preset algorithm to obtain multiple first weight values; each first valid word is sorted according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; a log parsing model is obtained, and the first sequence is matched by the log template tree in the log parsing model to obtain the classification result of the first log file, which solves the technical problem that the fault log is parsed and analyzed according to the word frequency in the related art, resulting in relatively low processing efficiency. The first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, thereby achieving the effect of improving the log processing efficiency.

[0117] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0119] Example 3

[0120] According to an embodiment of the present application, a flow chart of a method for classifying logs of a data center is also provided. Figure 7 As shown, the method includes:

[0121] Step 701: Receive a first log file to be classified from a data center sent by a client.

[0122] Specifically, the first log file that needs to be classified is sent to the cloud server through the client.

[0123] Step 702: Process the first log file in the cloud server to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; based on the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain multiple first weight values; sort each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; match the first sequence through the log template tree in the log parsing model to obtain a classification result of the first log file.

[0124] Specifically, the first log file is processed in the cloud server to obtain a first valid word set and a first word frequency corresponding to the first valid word; based on the first word frequency, a first weight value corresponding to each first valid word is calculated by a first preset algorithm, and then each first valid word is sorted according to multiple first weight values ​​to obtain a first sequence, and the first sequence is matched through a log template tree in a log parsing model to obtain a classification result of the first log file.

[0125] Step 703: Return the classification result to the client.

[0126] Using cloud servers to classify logs improves the efficiency of log classification and reduces the storage pressure on local terminals.

[0127] In the cloud server, the specific method for classifying the logs is the same as that in the first embodiment, and will not be described in detail here.

[0128] Example 4

[0129] According to an embodiment of the present application, a classification device for implementing the logs of the above data center is also provided, such as Figure 8 As shown, the device includes: a first processing unit 801, a first calculation unit 802, a second processing unit 803 and a matching unit 804.

[0130] The first processing unit 801 is used to process the first log file to be classified in the data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule.

[0131] The first calculation unit 802 is used to calculate the first weight value corresponding to each first valid word through a first preset algorithm according to the first word frequency corresponding to each first valid word, so as to obtain multiple first weight values.

[0132] The second processing unit 803 is used to sort each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word.

[0133] The matching unit 804 is used to obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0134] The third embodiment of the present invention provides a data center log classification device, which processes the first log file to be classified in the data center through the first processing unit 801 to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; the first calculation unit 802 calculates the first weight value corresponding to each first valid word through a first preset algorithm based on the first word frequency corresponding to each first valid word, and obtains multiple first weight values; the second processing unit 803 sorts each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; the matching unit 804 obtains a log parsing model, and matches the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file, which solves the technical problem of parsing and analyzing fault logs according to word frequency in the related art, resulting in relatively low processing efficiency. The first valid word of the first log file and the first weight value of the first valid word can be effectively and quickly matched with the log template tree in the log parsing model to obtain the classification result of the first log file, thereby achieving the effect of improving log processing efficiency.

[0135] Optionally, in the data center log classification device provided in Example 3 of the present invention, the first processing unit 801 includes: a cleaning module, used to perform data cleaning on the first log file through a preset regular expression to obtain a processed first log file; a statistical module, used to perform initial valid word statistics on the processed first log file to obtain multiple initial valid words; a first processing module, used to perform noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

[0136] Optionally, in the data center log classification device provided in Example 3 of the present invention, the first processing module includes: a screening submodule, used to identify and screen multiple initial valid words through a second preset algorithm to obtain a first valid word set; a processing submodule, used to perform noise reduction processing on the word frequency of each first valid word in the first valid word set through a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0137] Optionally, in the log classification device provided in the third embodiment of the present invention, the first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0138] Optionally, in the data center log classification device provided in Embodiment 3 of the present invention, the second processing unit 803 includes: a screening module, used to screen out first weight values ​​greater than a first preset value from multiple first weight values ​​to obtain multiple screened first weight values; a first sorting module, used to sort each first valid word according to the multiple screened first weight values ​​to obtain a first sequence.

[0139] Optionally, in the data center log classification device provided in Example 3 of the present invention, the device also includes: an updating unit, which is used to iteratively update the log parsing model according to the classification result of the first log file after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file.

[0140] Optionally, in the data center log classification device provided in the third embodiment of the present invention, the matching unit includes: a second processing module, which is used to obtain multiple sample log files, and process the sample log files to obtain multiple second valid word sets and second word frequencies corresponding to each second valid word in the second valid word sets, wherein the second valid words are the words remaining after the sample log files are processed according to the first preset rule; a calculation module, which is used to calculate the second weight value corresponding to each second valid word by a first preset algorithm according to the second word frequency corresponding to each second valid word, and obtain multiple second weight values; a second sorting module, which is used to sort the second valid words in each second valid word set according to the multiple second weight values, and obtain multiple second sequences, wherein the second sequence is composed of the second valid words of the sample log file and the second weight values ​​of the second valid words; a construction module, which is used to construct a template tree corresponding to the second sequence according to each second sequence, obtain multiple first log template trees, and save the multiple first log template trees to the template library; a building module, which is used to build the multiple log template trees in the template library to obtain a log template forest, and use the log template forest as a log parsing model.

[0141] Optionally, in the log classification device for a data center provided in Example 3 of the present invention, the construction module includes: a judgment submodule, used to judge whether there is a second log template tree in the template library that is identical to the root node of the second sequence; and a construction submodule, used to construct a first log template tree based on the second sequence if the second log template tree does not exist.

[0142] Optionally, in the log classification device provided in Embodiment 3 of the present invention, the device further includes: an insertion module, configured to insert the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree if there is a second log template tree.

[0143] Optionally, in the data center log classification device provided in the third embodiment of the present invention, the device also includes: a traversal unit, which is used to traverse each log template tree in the template library before assembling multiple log template trees in the template library to obtain a log template forest, so as to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; a pruning unit, which is used to prune the valid words in the multiple third sequences according to a second preset rule to obtain a processed third sequence; a sorting unit, which is used to sort the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; a second calculation unit, which is used to calculate the similarity of the multiple fourth sequences through a third preset algorithm to obtain multiple similarity coefficients; a screening unit, which is used to screen out similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; a third processing unit, which is used to merge the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and save the merged log template tree to the template library.

[0144] It should be noted that the first processing unit 801, the first calculation unit 802, the second processing unit 803 and the matching unit 804 correspond to steps S201 to S204 in Embodiment 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Embodiment 1.

[0145] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0146] Embodiment 3 of the present invention also provides a log classification system, which includes: a word frequency calculation processing module, a construction module and a classification module, wherein the word frequency calculation processing module is used to process the log file to obtain the corresponding valid words, the word frequency of the valid words and the weight value of the valid words. The construction module is used to construct the log template tree and the log template forest, and the classification module is used to implement the classification of the log file. The different modules of the system can be deployed in their respective appropriate computing platforms, which can improve the processing speed and capability of the log template parsing model.

[0147] Example 5

[0148] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0149] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0150] In this embodiment, the computer terminal can execute the following steps of the log classification method: process the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; calculate the first weight value corresponding to each first valid word by a first preset algorithm based on the first word frequency corresponding to each first valid word, and obtain multiple first weight values; sort each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0151] The computer terminal can also execute the program code of the following steps in the log classification method: processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain the first valid word set and the first word frequency corresponding to each first valid word in the first valid word set.

[0152] The above-mentioned computer terminal can also execute the program code of the following steps in the log classification method: performing noise reduction processing on multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: identifying and screening multiple initial valid words through a second preset algorithm to obtain a first valid word set; performing noise reduction processing on the word frequency of each first valid word in the first valid word set through a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0153] The computer terminal can also execute the program code of the following steps in the log classification method: the first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0154] The above-mentioned computer terminal can also execute the program code of the following steps in the log classification method: sorting each first valid word according to multiple first weight values ​​to obtain a first sequence, including: screening out a first weight value greater than a first preset value from multiple first weight values ​​to obtain multiple screened first weight values; sorting each first valid word according to the multiple screened first weight values ​​to obtain a first sequence.

[0155] The above-mentioned computer terminal can also execute the program code of the following steps in the log classification method: after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method also includes: iteratively updating the log parsing model based on the classification result of the first log file.

[0156] The computer terminal can also execute the program code of the following steps in the log classification method: obtaining the log parsing model includes: obtaining multiple sample log files, and processing the sample log files to obtain multiple second effective word sets and second word frequencies corresponding to each second effective word in the second effective word sets, wherein the second effective words are the words remaining after the sample log files are processed according to the first preset rule; according to the second word frequency corresponding to each second effective word, the second weight value corresponding to each second effective word is calculated by the first preset algorithm to obtain multiple second weight values; according to the multiple second weight values, the second effective words in each second effective word set are sorted to obtain multiple second sequences, wherein the second sequence is composed of the second effective words of the sample log file and the second weight values ​​of the second effective words; according to each second sequence, a template tree corresponding to the second sequence is constructed to obtain multiple first log template trees, and the multiple first log template trees are saved in the template library; the multiple log template trees in the template library are assembled to obtain a log template forest, and the log template forest is used as the log parsing model.

[0157] The computer terminal can also execute the program code of the following steps in the log classification method: constructing a template tree corresponding to the second sequence based on each second sequence to obtain multiple first log template trees, including: determining whether there is a second log template tree with the same root node as the second sequence in the template library; if there is no second log template tree, constructing a first log template tree based on the second sequence.

[0158] The computer terminal may also execute the program code of the following steps in the log classification method: if there is a second log template tree, inserting the second valid word in the second sequence and the second weight value corresponding to the second valid word into the second log template tree.

[0159] The computer terminal can also execute the program code of the following steps in the log classification method: before the multiple log template trees in the template library are assembled to obtain the log template forest, the method also includes: traversing each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; pruning the valid words in the multiple third sequences according to the second preset rule to obtain the processed third sequence; sorting the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculating the similarity of the multiple fourth sequences by the third preset algorithm to obtain multiple similarity coefficients; screening out the similarity coefficient greater than the second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merging the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and saving the merged log template tree to the template library.

[0160] Optionally, Fig. 9 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig. 9 As shown, the computer terminal 10 may include: one or more ( Fig. 9 Only one processor and memory are shown.

[0161] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the log classification method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned log classification method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: process the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; according to the first word frequency corresponding to each first valid word, calculate the first weight value corresponding to each first valid word through a first preset algorithm to obtain multiple first weight values; sort each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word; obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0163] Optionally, the processor may also execute the program code of the following steps: processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain the first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

[0164] Optionally, the processor may also execute the program code of the following steps: performing noise reduction processing on multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: identifying and screening multiple initial valid words through a second preset algorithm to obtain a first valid word set; performing noise reduction processing on the word frequency of each first valid word in the first valid word set through a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0165] Optionally, the processor may also execute program codes of the following steps: The first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0166] Optionally, the processor may also execute program code for the following steps: sorting each first valid word according to a plurality of first weight values ​​to obtain a first sequence, including: filtering out a first weight value greater than a first preset value from a plurality of first weight values ​​to obtain a plurality of filtered first weight values; sorting each first valid word according to the plurality of filtered first weight values ​​to obtain a first sequence.

[0167] Optionally, the processor may also execute program code of the following steps: after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method further includes: iteratively updating the log parsing model based on the classification result of the first log file.

[0168] Optionally, the processor may also execute the program code of the following steps: obtaining a log parsing model includes: obtaining a plurality of sample log files, and processing the sample log files to obtain a plurality of second valid word sets and a second word frequency corresponding to each second valid word in the second valid word sets, wherein the second valid word is the word remaining after the sample log file is processed according to the first preset rule; according to the second word frequency corresponding to each second valid word, calculating the second weight value corresponding to each second valid word by a first preset algorithm to obtain a plurality of second weight values; sorting the second valid words in each second valid word set according to the plurality of second weight values ​​to obtain a plurality of second sequences, wherein the second sequence is composed of the second valid words and the second weight values ​​of the second valid words in the sample log file; constructing a template tree corresponding to the second sequence according to each second sequence to obtain a plurality of first log template trees, and saving the plurality of first log template trees to a template library; assembling a plurality of log template trees in the template library to obtain a log template forest, and using the log template forest as a log parsing model.

[0169] Optionally, the processor may also execute the following program code: constructing a template tree corresponding to the second sequence based on each second sequence to obtain multiple first log template trees, including: determining whether there is a second log template tree in the template library that is identical to the root node of the second sequence; if there is no second log template tree, constructing a first log template tree based on the second sequence.

[0170] Optionally, the processor may further execute a program code of the following steps: if a second log template tree exists, inserting a second valid word in the second sequence and a second weight value corresponding to the second valid word into the second log template tree.

[0171] Optionally, the processor may also execute the following program code: before assembling multiple log template trees in the template library to obtain a log template forest, traverse each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; prune the valid words in the multiple third sequences according to a second preset rule to obtain a processed third sequence; sort the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculate the similarity of the multiple fourth sequences by a third preset algorithm to obtain multiple similarity coefficients; select similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merge the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and save the merged log template tree to the template library.

[0172] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig. 9 The structure of the electronic device is not limited. For example, the computer terminal 10 may also include Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations are shown.

[0173] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0174] Example 6

[0175] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the log classification method provided in the first embodiment.

[0176] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0177] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is the word remaining after the first log file is processed according to the first preset rule; based on the first word frequency corresponding to each first valid word, calculating the first weight value corresponding to each first valid word by a first preset algorithm to obtain multiple first weight values; sorting each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word; obtaining a log parsing model, and matching the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file.

[0178] The above-mentioned storage medium is also configured to store program codes for executing the following steps: processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: performing data cleaning on the first log file through a preset regular expression to obtain a processed first log file; performing initial valid word statistics on the processed first log file to obtain multiple initial valid words; performing noise reduction processing on the multiple initial valid words through a second preset algorithm to obtain the first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

[0179] The above-mentioned storage medium is also configured to store program codes for executing the following steps: performing noise reduction processing on multiple initial valid words through a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: identifying and screening multiple initial valid words through a second preset algorithm to obtain a first valid word set; performing noise reduction processing on the word frequency of each first valid word in the first valid word set through a second preset algorithm to obtain a first word frequency corresponding to each first valid word.

[0180] The storage medium is also configured to store program codes for executing the following steps: The first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

[0181] The above-mentioned storage medium is also configured to store program codes for executing the following steps: sorting each first valid word according to multiple first weight values ​​to obtain a first sequence, including: screening out first weight values ​​greater than a first preset value from multiple first weight values ​​to obtain multiple screened first weight values; sorting each first valid word according to the multiple screened first weight values ​​to obtain a first sequence.

[0182] The above-mentioned storage medium is also configured to store program codes for executing the following steps: after obtaining the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method also includes: iteratively updating the log parsing model based on the classification result of the first log file.

[0183] The storage medium is also configured to store program codes for executing the following steps: obtaining a log parsing model includes: obtaining a plurality of sample log files, and processing the sample log files to obtain a plurality of second effective word sets and a second word frequency corresponding to each second effective word in the second effective word sets, wherein the second effective word is a word remaining after the sample log files are processed according to a first preset rule; according to the second word frequency corresponding to each second effective word, a second weight value corresponding to each second effective word is calculated by a first preset algorithm to obtain a plurality of second weight values; according to the plurality of second weight values, the second effective words in each second effective word set are sorted to obtain a plurality of second sequences, wherein the second sequence is composed of the second effective words of the sample log file and the second weight values ​​of the second effective words; according to each second sequence, a template tree corresponding to the second sequence is constructed to obtain a plurality of first log template trees, and the plurality of first log template trees are saved in a template library; the plurality of log template trees in the template library are assembled to obtain a log template forest, and the log template forest is used as a log parsing model.

[0184] The storage medium is also configured to store program codes for executing the following steps: constructing a template tree corresponding to the second sequence based on each second sequence to obtain multiple first log template trees, including: determining whether there is a second log template tree with the same root node as the second sequence in the template library; if there is no second log template tree, constructing a first log template tree based on the second sequence.

[0185] The storage medium is also configured to store program codes for executing the following steps: if a second log template tree exists, inserting a second valid word in the second sequence and a second weight value corresponding to the second valid word into the second log template tree.

[0186] The storage medium is further configured to store program codes for executing the following steps: before assembling multiple log template trees in the template library to obtain a log template forest, traverse each log template tree in the template library to obtain multiple third sequences, wherein the third sequence is composed of valid words in each log template tree; prune the valid words in the multiple third sequences according to a second preset rule to obtain a processed third sequence; sort the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain multiple fourth sequences; calculate the similarity of the multiple fourth sequences by a third preset algorithm to obtain multiple similarity coefficients; select similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; merge the log template trees corresponding to each target similarity coefficient to obtain a merged log template tree, and save the merged log template tree to the template library.

[0187] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0188] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0193] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data center log classification method, characterized in that: include: Processing a first log file to be classified in a data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule; According to the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain a plurality of first weight values; Sorting each first valid word according to the plurality of first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word; Acquire a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file; According to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by a first preset algorithm, and the plurality of first weight values ​​obtained include: According to the first word frequency corresponding to each first effective word and the number of consecutive non-appearances of each first effective word, the first weight value corresponding to each first effective word is calculated by the first preset algorithm to obtain the multiple first weight values.

2. A log classification method, characterized in that: include: Processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule; According to the first word frequency corresponding to each first valid word, a first weight value corresponding to each first valid word is calculated by a first preset algorithm to obtain a plurality of first weight values; Sorting each first valid word according to the plurality of first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word; Acquire a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file; According to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by a first preset algorithm, and the plurality of first weight values ​​obtained include: According to the first word frequency corresponding to each first effective word and the number of consecutive non-appearances of each first effective word, the first weight value corresponding to each first effective word is calculated by the first preset algorithm to obtain the multiple first weight values.

3. The method according to claim 2, characterized in that Processing the first log file to be classified to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set includes: Performing data cleaning on the first log file by using a preset regular expression to obtain a processed first log file; Performing initial valid word statistics on the processed first log file to obtain a plurality of initial valid words; The plurality of initial valid words are subjected to noise reduction processing by using a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set.

4. The method according to claim 3, characterized in that The noise reduction process is performed on the multiple initial valid words by using a second preset algorithm to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, including: Identify and screen the multiple initial valid words using the second preset algorithm to obtain the first valid word set; The word frequency of each first valid word in the first valid word set is subjected to noise reduction processing by using the second preset algorithm to obtain a first word frequency corresponding to each first valid word.

5. The method according to claim 2, characterized in that: The first preset algorithm is in the form of: Among them, w i is the first weight value of the first valid word, p i is the frequency of each first valid word when the first log file was classified last time, T i is the first word frequency, α is a constant between 0 and 1, c is a constant between 0 and 7, x i is the number of times the valid word i does not appear consecutively.

6. The method according to claim 2, characterized in that Sorting each first valid word according to the plurality of first weight values ​​to obtain a first sequence includes: Filter out first weight values ​​greater than a first preset value from the plurality of first weight values ​​to obtain a plurality of filtered first weight values; Each first valid word is sorted according to the screened multiple first weight values ​​to obtain the first sequence.

7. The method according to claim 2, characterized in that After acquiring the log parsing model and matching the first sequence through the log template tree in the log parsing model to obtain the classification result of the first log file, the method further includes: The log parsing model is iteratively updated according to the classification result of the first log file.

8. The method according to claim 2, characterized in that: Obtaining the log parsing model includes: Acquire multiple sample log files, and process the sample log files to obtain multiple second effective word sets and second word frequencies corresponding to each second effective word in the second effective word sets, wherein the second effective words are words remaining after the sample log files are processed according to the first preset rule; According to the second word frequency corresponding to each second effective word, the second weight value corresponding to each second effective word is calculated by the first preset algorithm to obtain a plurality of second weight values; Sorting the second valid words in each of the second valid word sets according to the multiple second weight values ​​to obtain multiple second sequences, wherein the second sequences are composed of the second valid words of the sample log file and the second weight values ​​of the second valid words; Constructing a template tree corresponding to the second sequence according to each second sequence to obtain a plurality of first log template trees, and saving the plurality of first log template trees into a template library; A plurality of log template trees in the template library are assembled to obtain a log template forest, and the log template forest is used as the log parsing model.

9. The method according to claim 8, characterized in that Constructing a template tree corresponding to each second sequence according to the second sequence, obtaining a plurality of first log template trees including: Determine whether there is a second log template tree in the template library that is identical to the root node of the second sequence; If the second log template tree does not exist, construct the first log template tree according to the second sequence.

10. The method according to claim 9, characterized in that The method further comprises: If the second log template tree exists, the second valid word in the second sequence and the second weight value corresponding to the second valid word are inserted into the second log template tree.

11. The method according to claim 8, characterized in that Before assembling a plurality of log template trees in the template library to obtain a log template forest, the method further includes: Traversing each log template tree in the template library to obtain a plurality of third sequences, wherein the third sequence is composed of valid words in each log template tree; Performing pruning processing on the valid words in the plurality of third sequences according to a second preset rule to obtain a processed third sequence; sorting the valid words in the processed third sequence according to the order of the first letters of the valid words in the processed third sequence to obtain a plurality of fourth sequences; Performing similarity calculation on the plurality of fourth sequences by using a third preset algorithm to obtain a plurality of similarity coefficients; Filtering out similarity coefficients greater than a second preset value from the multiple similarity coefficients to obtain multiple target similarity coefficients; The log template trees corresponding to each target similarity coefficient are merged to obtain a merged log template tree, and the merged log template tree is saved in the template library.

12. A data center log classification method, characterized in that: include: receiving a first log file to be classified from a data center sent by a client; The first log file is processed in the cloud server to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule, and a first weight value corresponding to each first valid word is calculated by a first preset algorithm according to the first word frequency corresponding to each first valid word to obtain a plurality of first weight values; each first valid word is sorted according to the plurality of first weight values ​​to obtain a first sequence, wherein the first sequence is composed of each first valid word and the first weight value corresponding to each first valid word, and the first sequence is matched by a log template tree in a log parsing model to obtain a classification result of the first log file; Returning the classification result to the client; According to the first word frequency corresponding to each first valid word, the first weight value corresponding to each first valid word is calculated by a first preset algorithm, and the plurality of first weight values ​​obtained include: According to the first word frequency corresponding to each first effective word and the number of consecutive non-appearances of each first effective word, the first weight value corresponding to each first effective word is calculated by the first preset algorithm to obtain the multiple first weight values.

13. A data center log classification device, characterized in that: include: A first processing unit is used to process a first log file to be classified in a data center to obtain a first valid word set and a first word frequency corresponding to each first valid word in the first valid word set, wherein the first valid word is a word remaining after the first log file is processed according to a first preset rule; A first calculation unit, configured to calculate a first weight value corresponding to each first valid word by a first preset algorithm according to a first word frequency corresponding to each first valid word, to obtain a plurality of first weight values; A second processing unit is used to sort each first valid word according to the multiple first weight values ​​to obtain a first sequence, wherein the first sequence consists of each first valid word and the first weight value corresponding to each first valid word; a matching unit, configured to obtain a log parsing model, and match the first sequence through a log template tree in the log parsing model to obtain a classification result of the first log file; Wherein, the classification device is also used to: calculate the first weight value corresponding to each first effective word through the first preset algorithm according to the first word frequency corresponding to each first effective word and the number of consecutive non-appearances corresponding to each first effective word, to obtain the multiple first weight values.

14. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the method for classifying logs of a data center described in claim 1, or the method for classifying logs described in any one of claims 2 to 11.

Images