Log processing method, device, computer system and readable storage medium

Abstract

The present disclosure provides a log processing method, which can be applied to technical fields such as big data, artificial intelligence, and the Internet. The method includes: determining target log data in a log to be processed; performing part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; based on the part-of-speech tagging sequence and the target log data, determining a target template log that matches the log to be processed from multiple template logs, wherein each of the multiple template logs is a log with a known category result; and determining a category result of the log to be processed based on the category result of the target template log. The present disclosure also provides a log processing device, a computer system, a readable storage medium, and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as big data, artificial intelligence, and the Internet, and more specifically, to a log processing method, device, computer system, readable storage medium, and computer program product. Background Art

[0002] During the operation of application operation and maintenance services, a large amount of logs are generated every day, which record the operation status of application operation and maintenance services and other information. By analyzing the logs to monitor the operation status of application operation and maintenance services, problems can be discovered and solved in a timely manner.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: low processing efficiency and low recognition accuracy when performing real-time analysis on operation and maintenance logs. Summary of the invention

[0004] In view of this, the present disclosure provides a log processing method, device, computer system, readable storage medium and computer program product.

[0005] One aspect of the present disclosure provides a log processing method, including: determining target log data in a log to be processed; performing part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; based on the part-of-speech tagging sequence and the target log data, determining a target template log that matches the log to be processed from multiple template logs, wherein each of the multiple template logs is a log with a known category result; and determining a category result of the log to be processed based on the category result of the target template log.

[0006] Another aspect of the present disclosure provides a log processing device, including: a data determination module, used to determine target log data in a log to be processed; a part-of-speech determination module, used to perform part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; a template determination module, used to determine a target template log matching the log to be processed from multiple template logs based on the part-of-speech tagging sequence and the target log data, wherein each of the multiple template logs is a log with a known category result; and a result determination module, used to determine a category result of the log to be processed based on the category result of the target template log.

[0007] Another aspect of the present disclosure provides a computer system, comprising:

[0008] one or more processors;

[0009] a memory for storing one or more programs,

[0010] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0011] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0012] Another aspect of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0013] According to an embodiment of the present disclosure, a log processing method is adopted, including: determining target log data in a log to be processed; performing part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; determining a target template log that matches the log to be processed from multiple template logs based on the part-of-speech tagging sequence and the target log data, wherein each of the multiple template logs is a log of a known category result; and a technical means for determining the category result of the log to be processed based on the category result of the target template log, determining the target template log from multiple template logs based on the part-of-speech tagging sequence and the target log data, introducing part-of-speech analysis of the target log data, thereby improving matching accuracy so that operation and maintenance personnel can accurately locate the source of the problem and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 An exemplary system architecture to which the log processing method and device disclosed herein can be applied is schematically shown;

[0016] Figure 2 The following schematically shows an application scenario diagram of the log processing method according to an embodiment of the present disclosure;

[0017] Figure 3 The flowchart of the log processing method according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 4 A flowchart of a log processing method according to another embodiment of the present disclosure is schematically shown;

[0019] Figure 5 Schematically shows a flow chart of determining a target template log according to another embodiment of the present disclosure;

[0020] Figure 6 A block diagram schematically shows a log processing device according to an embodiment of the present disclosure; and

[0021] Figure 7 A block diagram of a computer system suitable for implementing a log processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0025] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] The present disclosure provides a log processing method, device, computer system, readable storage medium and computer program product.

[0027] According to an embodiment of the present disclosure, a log processing method may include: determining target log data in a log to be processed; performing part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; based on the part-of-speech tagging sequence and the target log data, determining a target template log that matches the log to be processed from multiple template logs, wherein each of the multiple template logs is a log with a known category result; and determining a category result of the log to be processed based on the category result of the target template log.

[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0029] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] Figure 1 The exemplary system architecture 100 to which the log processing method and apparatus according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as log processing applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0033] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0034] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by the terminal devices 101, 102, and 103. The background management server may analyze and process the received application-level logs, and feed back the processing results (such as whether the logs show alarms, etc.) to the operation and maintenance engineer to accurately locate the problem.

[0035] It should be noted that the log processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the log processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The log processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the log processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] During the operation of application operation and maintenance services, a large amount of logs are generated every day, which record the operation status information of network systems and application operation and maintenance services, etc. By analyzing the logs to monitor the operation status of network systems and application operation and maintenance services, problems can be discovered and solved in a timely manner.

[0038] Figure 2 The application scenario diagram of the log processing method according to an embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the received log is used as the log to be processed 210, and the log processing method provided by the embodiment of the present disclosure is performed to determine the category result of the log to be processed, such as the category result characterizing the normal operation status, or the category result characterizing the abnormal operation status. The log to be processed with the category result characterizing the normal operation status can be stored in the first database 220 for archiving. The log to be processed with the category result characterizing the abnormal operation status can be stored in the second database 230. For the log to be processed with the category result characterizing the abnormal operation status, the operation and maintenance personnel can determine the specific operation and maintenance application problems based on the category results, such as database, middleware or host problems. The operation and maintenance personnel can solve them in a more targeted manner according to the category results.

[0040] Figure 3The flowchart of the log processing method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 3 As shown, the method includes operations S310 to S340.

[0042] In operation S310 , target log data in a log to be processed is determined.

[0043] In operation S320 , part-of-speech analysis is performed on the target log data to determine a part-of-speech tagging sequence of the target log data.

[0044] In operation S330 , based on the part-of-speech tag sequence and the target log data, a target template log matching the log to be processed is determined from a plurality of template logs, wherein each of the plurality of template logs is a log of a known category result.

[0045] In operation S340 , based on the category result of the target template log, a category result of the log to be processed is determined.

[0046] According to the embodiments of the present disclosure, the amount of target log data is not limited, and it can be multiple keywords extracted from the log to be processed, or multiple text paragraphs. The type of target log data is not limited, and it can be a vector or text such as Chinese, English, or characters. As long as it is data extracted from the log to be processed, it can be used as target log data.

[0047] According to an embodiment of the present disclosure, part-of-speech analysis can be performed on the target log data to determine a part-of-speech tagging sequence corresponding to the target log data. For example, taking the target log data as a plurality of keywords, each keyword can represent a type of part-of-speech, such as a verb, a noun, an adjective, etc. A part-of-speech tagging sequence corresponding to the target log data can be obtained according to the arrangement order of the plurality of keywords.

[0048] According to an embodiment of the present disclosure, multiple template logs can be pre-set, and the category results of the multiple template logs are different from each other. The multiple template logs can be mapped one by one to the multiple category results to establish a mapping relationship. Based on the part-of-speech tagging sequence and the target log data, a target template log matching the log to be processed is determined from the multiple template logs, and the category result of the target template log is determined according to the mapping relationship, and then the category result of the log to be processed is determined.

[0049] According to other embodiments of the present disclosure, the target template log can be determined from multiple template logs by log query, fuzzy matching, or keyword matching based only on the target log data, and then the category result of the log to be processed is determined based on the category result of the target template log. However, by determining the target template log from multiple template logs based on the part-of-speech tag sequence and the target log data, the part-of-speech analysis of the target log data is introduced, which can improve the matching accuracy, so that the operation and maintenance personnel can accurately locate the source of the problem, and improve the processing efficiency.

[0050] By using the log processing method provided by the embodiment of the present disclosure, logs with abnormal operation and maintenance status can be captured in time, and then timely and effectively processed based on the classification results, thereby avoiding the problem of long time consumption and weak migration capability caused by using rules for identification.

[0051] Reference below Figure 4-5 , combined with specific embodiments Figure 3 The method shown is further explained.

[0052] Figure 4 The flowchart of a log processing method according to another embodiment of the present disclosure is schematically shown.

[0053] like Figure 4 As shown, the log 410 to be processed is input into the tag tree model 420 to determine the log classification tag 430 that matches the log 410 to be processed. Using the tree structure, the log 410 to be processed is converted into the initial log data 440 that matches the log classification tag 430. The initial log data 440 is filtered for stop words and template words to obtain the target log data 450. The target log data 450 in the log 410 to be processed is subjected to part-of-speech analysis to determine the part-of-speech tag sequence 460 of the target log data. Based on the part-of-speech tag sequence 460 and the target log data 450, a target template log 471 that matches the log 410 to be processed is determined from multiple template logs 470. Each of the multiple template logs 470 is a log with a known category result. Based on the category result of the target template log 471, the category result 480 of the log 410 to be processed is determined.

[0054] According to an embodiment of the present disclosure, log classification labels may include application-class log classification labels, for example, log classification labels related to databases, log classification labels related to caches, log classification labels related to message queues, log classification labels related to processors, and log classification labels related to networks, etc.

[0055] According to an embodiment of the present disclosure, the category results of multiple template logs may be different, but the multiple template logs may belong to the same log category label, for example, to a log category label related to a database. The multiple category results may be respectively an abnormal category result representing an incorrect connection password, an abnormal category result representing an abnormal query interruption, an abnormal category result representing a timeout in waiting for a lock, and an abnormal category result representing an inability to connect to a database.

[0056] According to an embodiment of the present disclosure, a label tree model can be used to perform preliminary screening of logs to be processed and determine log classification labels of logs to be processed so as to process data in the logs to be processed. For example, based on the log classification labels, multiple template logs that match the log classification labels of logs to be processed can be determined from a template log set, thereby simplifying the amount of subsequent processing calculations and improving the accuracy of matching the logs to be processed with the target template logs.

[0057] For example, when an application alarm log is received as a log to be processed, the log classification label of the log to be processed can be determined based on the label tree model, such as a network-related log classification label. Based on the network-related log classification label, multiple template logs matching the log classification label are determined from the template log set, and the target log data and part-of-speech tag sequence can be used to determine the target template log from multiple template logs. Based on the target template log, the category result of the log to be processed is determined, thereby achieving the sub-classification of the log to be processed, and finally identifying the category result of the log to be processed as an abnormal category result for characterizing "http connection timeout".

[0058] According to an embodiment of the present disclosure, a label tree model may refer to a model that applies label tree logic, and may be obtained by training based on log sample data and predetermined log classification labels as training samples. All data of the log to be processed may be input into the label tree model to obtain a log classification label that matches the log to be processed.

[0059] According to the embodiments of the present disclosure, a label tree model is used to implement rapid clustering of logs to be processed, which can improve the accuracy and effectiveness of the initial classification.

[0060] According to an embodiment of the present disclosure, a tree structure matching a log classification label can be used to convert a log to be processed into initial log data matching the log classification label. The tree structure may include nodes and paths. A node may include a word in the log to be processed, and a path may refer to an edge used to connect each node. Based on the nodes in the tree structure, a word matching a node can be determined from the log to be processed as a keyword. The arrangement order of multiple keywords is determined according to the path, and then initial log data that meets the template standard is generated.

[0061] According to the embodiments of the present disclosure, by utilizing the tag tree model and the tree structure, keywords can be extracted and corrected for the log to be processed, thereby accurately extracting keywords and generating a format that matches the template log, thereby simplifying subsequent processing operations.

[0062] According to an embodiment of the present disclosure, a template word set may refer to content shared by multiple template logs. For example, content that appears in multiple template logs and cannot be used to distinguish whether the operation and maintenance status is abnormal or normal may be called a template word set.

[0063] According to an embodiment of the present disclosure, a stop word may refer to an invalid word, such as a punctuation mark, an invalid identifier, and the like.

[0064] According to an embodiment of the present disclosure, the removal of stop words and template words in the initial log data may be performed in steps, but the execution order is not limited.

[0065] According to the embodiments of the present disclosure, a stop word library and a template word library can be pre-built. When performing the operation of filtering the initial log data for stop words and template word libraries, the words in the stop word library and the template word library can be used to match the content in the initial log data, and the content in the initial log data that matches the words in the stop word library and the template word library can be deleted, and the keywords are retained, which can be used as the target log data.

[0066] According to the embodiments of the present disclosure, the use of pre-established stop word library and template word library can reduce the heavy filtering tasks, improve the filtering coverage, and thus improve the troubleshooting efficiency.

[0067] According to other embodiments of the present disclosure, the operation of filtering the initial log data for stop words and template words may not be performed, and the initial log data may be used as the target log data.

[0068] According to an embodiment of the present disclosure, compared with taking the initial log data as the target log data, taking the filtered log data as the target log data, stop words and template words can be filtered, invalid words can be removed, and keywords can be retained, thereby reducing impurities in the log to be processed and eliminating noise, thereby reducing the complexity of the log processing method while improving the accuracy of the category results.

[0069] According to an embodiment of the present disclosure, with respect to operation S330, determining a target template log matching the log to be processed from multiple template logs based on the part-of-speech tag sequence and the target log data may include the following operations.

[0070] For example, based on the part-of-speech tag sequence and the target log data, the similarity between each of the multiple template logs and the log to be processed is determined to obtain multiple similarity results. The multiple similarity results correspond to the multiple template logs one by one. Based on the multiple similarity results, the target template log is determined from the multiple template logs.

[0071] According to the embodiment of the present disclosure, after obtaining multiple similarity results, the multiple similarity results can be sorted in descending order, and the template log with the highest similarity result is determined as the target template log. However, this method is not limited to this method. As long as the template log that is most similar to the log to be processed can be determined from multiple similarity results, it will be fine.

[0072] According to an embodiment of the present disclosure, a template log that is most similar to a log to be processed may be used as a target template log, and the category result of the target template log may be used as the category result of the log to be processed.

[0073] Figure 5 A flowchart for determining a target template log according to another embodiment of the present disclosure is schematically shown.

[0074] like Figure 5 As shown, the target log data 510 can be input into the part-of-speech analysis model 520 to obtain a part-of-speech tag sequence 530. Based on the part-of-speech tag sequence 530, the weight data 540 corresponding to the target log data is determined. Based on the weight data 540 and the target log data 510, the similarity between each of the multiple template logs 550 and the log to be processed is determined to obtain multiple similarity results. The multiple similarity results correspond to the multiple template logs one by one. Based on the multiple similarity results, the target template log 551 is determined from the multiple template logs 550.

[0075] According to an embodiment of the present disclosure, a part-of-speech analysis model may include a cascaded encoder, a bidirectional long short-term memory network (Bi-LSTM, Bi-Long Short-Term Memory) and a conditional random field (CRF, Conditional Random Fields), but is not limited to this. The part-of-speech analysis model may also include a cascaded encoder, a long short-term memory network and a conditional random field, or the part-of-speech analysis model may also only include an encoder and a conditional random field.

[0076] According to the embodiments of the present disclosure, the type of encoder is not limited. For example, it can be a BERT (Bidirectional Encoder Representation from Transformers) encoder, an ERNIE (Enhanced Language Representation with Informative Entities) encoder, or a Log2vec encoder, as long as it can encode keywords into a feature vector network.

[0077] According to an embodiment of the present disclosure, determining weight data corresponding to target log data based on a part-of-speech tagging sequence may include: determining confidence data matching the part-of-speech tagging sequence based on the part-of-speech tagging sequence; and determining weight data corresponding to the target log data based on the confidence data.

[0078] According to an embodiment of the present disclosure, the target log data is input into a part-of-speech analysis model to obtain a part-of-speech tagging sequence. For each part-of-speech tag in the part-of-speech tagging sequence, it can be determined by confidence data corresponding to multiple part-of-speech tags one by one. For example, the target log data may include multiple target log sub-data, and for the target log sub-data "connection", its output result may include the confidence data of each of the multiple part-of-speech tags, such as the confidence data 0.1 of the part-of-speech tag representing the noun, the confidence data 0.8 of the part-of-speech tag representing the verb, and the confidence data 0.1 of the part-of-speech tag representing the adjective. The weight data for the target sub-data "connection" can be determined based on the confidence data of the predetermined part-of-speech tag, for example, it can be 0.8, and the final part-of-speech tag is "verb". For another example, for the target log sub-data "time", its output result may include the confidence data 0.8 of the part-of-speech tag representing the noun, the confidence data 0.1 of the part-of-speech tag representing the verb, and the confidence data 0.1 of the part-of-speech tag representing the adjective. In view of this, the weight data for the target log sub-data "time" may be determined based on the confidence data of a predetermined part-of-speech tag, such as a verb part-of-speech tag, for example, 0.1, and the final part-of-speech tag is "noun".

[0079] According to other embodiments of the present disclosure, a mapping rule between part-of-speech tags and weight data may be pre-set, and weight data corresponding to the part-of-speech tag sequence may be determined based on the part-of-speech tag sequence and the mapping rule. For example, for each part-of-speech tag in the part-of-speech tag sequence, the weight data corresponding to the part-of-speech tag of "noun" may be predetermined to be 0.2, the weight data corresponding to the part-of-speech tag of "verb" may be predetermined to be 0.8, and so on.

[0080] According to the embodiments of the present disclosure, a part-of-speech tagging sequence is introduced to combine with the target log data to analyze the category results of the logs to be processed, and weight data is determined more specifically according to the part-of-speech tagging sequence, and the keywords of the parts of speech such as nouns, verbs, and adjectives are distinguished by the weight data. By adjusting the weights of key part-of-speech data such as verbs in the logs in the operation and maintenance field, the healthy logs representing the normal operation status and the abnormal logs representing the abnormal operation status can be distinguished more accurately from multiple logs.

[0081] According to an embodiment of the present disclosure, based on weight data and target log data, determining the similarity between each of a plurality of template logs and a log to be processed, and obtaining a plurality of similarity results may include the following operations.

[0082] For example, the target log data includes multiple target log sub-data, and the weight data includes multiple weight sub-data, and the multiple weight sub-data correspond to the multiple target log sub-data one by one. i , each keyword word i With a weight sub-data w i Correspondingly, i may be a positive integer from 1, 2, 3 to 1. I may be determined according to the number of target log sub-data, for example, I is equal to the number of target log sub-data.

[0083] For each target log sub-data in the multiple target log sub-data, the target log sub-data and the weight sub-data can be processed by using the word frequency and inverse text frequency index algorithm to determine the similarity between the target log sub-data and each of the multiple template logs, thereby obtaining multiple similarity sub-results of the target log sub-data and multiple similarity sub-result sets. j , where j can be a positive integer from 1, 2, 3 to J. J can be determined according to the number of template logs, for example, J is equal to the number of template logs. The multiple similarity sub-results correspond to the multiple template logs one by one, each similarity sub-result set in the multiple similarity sub-result sets includes multiple similarity sub-results of the target log sub-data that matches the similarity sub-result set, and the multiple similarity sub-result sets correspond to the multiple target log sub-data one by one.

[0084] According to an embodiment of the present disclosure, the term frequency-inverse document frequency (TF-IDF) algorithm can be as shown in formula (1).

[0085]

[0086] Among them, TF-IDF (word i) represents the target log sub-data word i The similarity sub-result between the template log Rj, w i represents the weight sub-data, J represents the total number of multiple template logs, count(template log R j Chinese word i ) represents the template log R j Chinese word i The number of words.

[0087] According to an embodiment of the present disclosure, for each template log in a plurality of template logs, a plurality of target similarity sub-results matching the template log are determined from a plurality of similarity sub-result sets. Based on the plurality of target similarity sub-results, the similarity between the template log and the log to be processed is determined, and a similarity result matching the template log is obtained, thereby obtaining a plurality of similarity results.

[0088] According to an embodiment of the present disclosure, the similarity result between the to-be-processed log and the template log can be determined by the following formula (2).

[0089]

[0090] Among them, TF-IDF (R j ) represents the log to be processed and the template log R j The similarity results of the matches.

[0091] According to the embodiments of the present disclosure, the word frequency and inverse text frequency index algorithms can be used to quickly and effectively determine the similarity between the log to be processed and multiple template logs, thereby determining the target template log. At the same time, weight data determined based on the part-of-speech tagging sequence is introduced to make the similarity results more accurate and effective.

[0092] According to the embodiments of the present disclosure, in actual operation, new stop words and new template word sets that are not in the stop word library and the template word set library can be determined in real time from the log to be processed, and the new stop words can be stored in the stop word library in time, and the new template word sets can be added to the template word set library, so that the template word set library and the stop word library can be updated in time. The template word set library and the stop word library are effectively generated incrementally, thereby improving the coverage of the template word set library and the stop word library.

[0093] According to the embodiment of the present disclosure, in actual operation, unclassified tags can also be added to unrecognizable logs to be processed. The logs to be processed without classification tags are stored for subsequent manual recognition, so that after processing, a template log with a new category result is generated.

[0094] Figure 6 The block diagram of a log processing device according to an embodiment of the present disclosure is schematically shown.

[0095] like Figure 6 As shown, the log processing device 600 includes a data determination module 610 , a part-of-speech determination module 620 , a template determination module 630 , and a result determination module 640 .

[0096] The data determination module 610 is used to determine target log data in the log to be processed.

[0097] The part-of-speech determination module 620 is used to perform part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data.

[0098] The template determination module 630 is used to determine a target template log matching the log to be processed from multiple template logs based on the part-of-speech tag sequence and the target log data, wherein each of the multiple template logs is a log of a known category result.

[0099] The result determination module 640 is used to determine the category result of the log to be processed based on the category result of the target template log.

[0100] According to an embodiment of the present disclosure, the template determination module 630 includes a similarity determination submodule and a template determination submodule.

[0101] The similarity determination submodule is used to determine the similarity between each of the multiple template logs and the log to be processed based on the part-of-speech tag sequence and the target log data, and obtain multiple similarity results, wherein the multiple similarity results correspond to the multiple template logs one by one.

[0102] The template determination submodule is used to determine a target template log from multiple template logs based on multiple similarity results.

[0103] According to an embodiment of the present disclosure, the similarity determination submodule includes a weight determination unit and a similarity determination unit.

[0104] The weight determination unit is used to determine weight data corresponding to the target log data based on the part-of-speech tag sequence.

[0105] The similarity determination unit is used to determine the similarity between each of the multiple template logs and the log to be processed based on the weight data and the target log data, and obtain multiple similarity results.

[0106] According to an embodiment of the present disclosure, the target log data includes a plurality of target log sub-data, the weight data includes a plurality of weight sub-data, and the plurality of weight sub-data correspond one-to-one to the plurality of target log sub-data.

[0107] According to an embodiment of the present disclosure, the similarity determination unit includes a first similarity determination subunit, a second similarity determination subunit, and a third similarity determination subunit.

[0108] The first similarity determination subunit is used to process the target log subdata and the weight subdata by using the word frequency and inverse text frequency index algorithm for each target log subdata in the multiple target log subdata, determine the similarity between the target log subdata and each of the multiple template logs, obtain multiple similarity subresults of the target log subdata, and obtain multiple similarity subresult sets, wherein the multiple similarity subresults correspond one-to-one to the multiple template logs, each similarity subresult set in the multiple similarity subresult sets includes multiple similarity subresults of the target log subdata that matches the similarity subresult set, and the multiple similarity subresult sets correspond one-to-one to the multiple target log subdata.

[0109] The second similarity determination subunit is used to determine, for each template log in the plurality of template logs, from the plurality of similarity sub-result sets, a plurality of target similarity sub-results that match the template log.

[0110] The third similarity determination subunit is used to determine the similarity between the template log and the log to be processed based on multiple target similarity sub-results, obtain a similarity result matching the template log, and obtain multiple similarity results.

[0111] According to an embodiment of the present disclosure, the weight determination unit includes a confidence determination subunit and a weight determination subunit.

[0112] The confidence determination subunit is used to determine the confidence data matching the part-of-speech tag sequence based on the part-of-speech tag sequence.

[0113] The weight determination subunit is used to determine weight data corresponding to the target log data based on the confidence data.

[0114] According to an embodiment of the present disclosure, the data determination module includes an input unit, a conversion unit, and a filtering unit.

[0115] The input unit is used to input the log to be processed into the label tree model to determine the log classification label matching the log to be processed.

[0116] The conversion unit is used to convert the log to be processed into initial log data matching the log classification label by using the tree structure.

[0117] The filtering unit is used to filter the initial log data by using stop words and template words to obtain log data.

[0118] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0119] For example, any multiple of the data determination module 610, the part of speech determination module 620, the template determination module 630, and the result determination module 640 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more modules / units / subunits in these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the data determination module 610, the part of speech determination module 620, the template determination module 630, and the result determination module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, at least one of the data determination module 610, the part-of-speech determination module 620, the template determination module 630, and the result determination module 640 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0120] It should be noted that the log processing device part in the embodiment of the present disclosure corresponds to the log processing method part in the embodiment of the present disclosure. The description of the log processing device part specifically refers to the log processing method part, which will not be repeated here.

[0121] Figure 7 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 7 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0122] like Figure 7 As shown, the computer system 700 according to the embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0123] In RAM 703, various programs and data required for the operation of system 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other through bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0124] According to an embodiment of the present disclosure, the system 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The system 700 may further include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0125] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0126] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0127] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0128] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0129] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains a program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the log processing method provided by the embodiment of the present disclosure.

[0130] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0132] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0134] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A log processing method, comprising: Determine target log data in the log to be processed; Performing part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; Based on the part-of-speech tag sequence and the target log data, determining a target template log matching the log to be processed from a plurality of template logs matching the log classification labels, wherein each of the plurality of template logs is a log of a known category result; and Based on the category result of the target template log, determining the category result of the log to be processed; Wherein, determining the target log data in the log to be processed includes: Inputting the log to be processed into a label tree model to determine a log classification label that matches the log to be processed; Using a tree structure matching the log classification label, converting the log to be processed into initial log data that meets the template standard; and The initial log data is filtered for stop words and template words to obtain the target log data.

2. The method according to claim 1, wherein: The step of determining a target template log matching the log to be processed from a plurality of template logs based on the part-of-speech tag sequence and the target log data comprises: Based on the part-of-speech tag sequence and the target log data, determine the similarity between each of the multiple template logs and the log to be processed, and obtain multiple similarity results, wherein the multiple similarity results correspond to the multiple template logs one by one; and Based on the multiple similarity results, the target template log is determined from the multiple template logs.

3. The method according to claim 2, wherein: The determining, based on the part-of-speech tag sequence and the target log data, the similarity between each of the multiple template logs and the log to be processed, and obtaining multiple similarity results includes: Determining weight data corresponding to the target log data based on the part-of-speech tag sequence; and Based on the weight data and the target log data, the similarity between each of the plurality of template logs and the log to be processed is determined to obtain the plurality of similarity results.

4. The method according to claim 3, wherein: The target log data includes a plurality of target log sub-data, the weight data includes a plurality of weight sub-data, and the plurality of weight sub-data correspond to the plurality of target log sub-data one by one; The determining, based on the weight data and the target log data, the similarity between each of the plurality of template logs and the log to be processed, and obtaining the plurality of similarity results comprises: For each target log sub-data in the multiple target log sub-data, use a word frequency and inverse text frequency index algorithm to process the target log sub-data and the weight sub-data, determine the similarity between the target log sub-data and each of the multiple template logs, obtain multiple similarity sub-results of the target log sub-data, and obtain multiple similarity sub-result sets, wherein the multiple similarity sub-results correspond to the multiple template logs one-to-one, each similarity sub-result set in the multiple similarity sub-result sets includes multiple similarity sub-results of the target log sub-data that matches the similarity sub-result set, and the multiple similarity sub-result sets correspond to the multiple target log sub-data one-to-one; For each template log in the plurality of template logs, determining a plurality of target similarity sub-results matching the template log from the plurality of similarity sub-result sets; and Based on the multiple target similarity sub-results, the similarity between the template log and the log to be processed is determined, a similarity result matching the template log is obtained, and the multiple similarity results are obtained.

5. The method according to claim 3 or 4, wherein: The determining, based on the part-of-speech tag sequence, weight data corresponding to the target log data comprises: Based on the part-of-speech tag sequence, determining confidence data matching the part-of-speech tag sequence; and Based on the confidence data, weight data corresponding to the target log data is determined.

6. A log processing device, comprising: A data determination module, used to determine target log data in the log to be processed; A part-of-speech determination module, used to perform part-of-speech analysis on the target log data to determine a part-of-speech tagging sequence of the target log data; a template determination module, configured to determine, based on the part-of-speech tag sequence and the target log data, a target template log matching the log to be processed from a plurality of template logs matching the log classification tags, wherein each of the plurality of template logs is a log of a known category result; and A result determination module, used to determine the category result of the log to be processed based on the category result of the target template log; Wherein, the data determination module includes: An input unit, used for inputting the log to be processed into a label tree model to determine a log classification label matching the log to be processed; A conversion unit, configured to convert the log to be processed into initial log data conforming to a template standard by using a tree structure matching the log classification label; The filtering unit is used to filter the stop words and template words of the initial log data to obtain the target log data.

7. A computer system comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, wherein when the instructions are executed by a processor, the processor implements the method according to any one of claims 1 to 5.

9. A computer program product comprising: Computer executable instructions, when executed, are used to implement the method according to any one of claims 1 to 5.

Images