Log data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a log data processing method, which can be applied to the fields of artificial intelligence technology and financial technology. The log data processing method includes: obtaining log data, wherein the log data includes at least one log record; vectorizing each log record to obtain a vectorized log set containing a log record vector; clustering the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records; identifying the log records in the same log cluster to obtain named entities; and generating a log template for the log cluster based on the named entity, wherein the log template is used to characterize the log structure characteristics of the log cluster. The present disclosure also provides a log data processing device, equipment, storage medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology and financial technology, and more specifically to a log data processing method, device, equipment, medium and program product. Background Art

[0002] Logs are data generated by information systems, recording the operating status of devices, operating systems, and application software, reflecting the true internal status of information systems, and are an important basis for information operation and maintenance managers and security managers to troubleshoot and analyze attack sources in daily operations. Therefore, the analysis and management of logs are becoming increasingly important. In related technologies, logs are classified and identified, classified by format, matched with corresponding data field processing paradigm rules, fields are extracted, and values ​​are assigned to specified fields to extract data.

[0003] In the process of implementing the technical solution disclosed in the present invention, the inventors found that there are at least the following problems in the related technology: the method in the related technology needs to pre-define the log format and adapt each log format. When the log format changes, format errors will occur in identifying the new log, resulting in log recognition failure. Technical personnel are required to re-establish data format matching rules to solve the problem. The entire log analysis and processing process is time-consuming, labor-intensive, and inefficient. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a log processing method, apparatus, device, medium and program product for improving log recognition speed.

[0005] According to a first aspect of the present disclosure, a log data processing method is provided, comprising:

[0006] Obtaining log data, wherein the log data includes at least one log record;

[0007] Performing vectorization processing on each of the above log records to obtain a vectorized log set containing log record vectors;

[0008] performing clustering processing on the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records;

[0009] Identifying the log records in the same log cluster to obtain named entities; and

[0010] A log template of the log cluster is generated according to the named entity, wherein the log template is used to characterize log structure features of the log cluster.

[0011] According to an embodiment of the present disclosure, the vectorization processing of each of the above-mentioned log records to obtain a vectorized log set containing log record vectors includes:

[0012] Preprocess each of the above log records;

[0013] Perform word segmentation and stop word removal on the preprocessed log records to obtain a word corpus;

[0014] Input the above word corpus into the vectorization model, vectorize the words, and output a word vector library; wherein the above word vector library includes the word vector corresponding to each of the above words;

[0015] According to the words corresponding to each of the log records and the word vectors corresponding to the words, a log record vector corresponding to each of the log records is determined to obtain the vectorized log set.

[0016] According to an embodiment of the present disclosure, clustering the log record vectors in the vectorized log set to form different log clusters includes:

[0017] Clustering the log record vectors in the vectorized log set to form different vector clusters;

[0018] The log record corresponding to each log record vector is determined according to the log record vectors in the same vector cluster to form the log cluster corresponding to the vector cluster.

[0019] According to an embodiment of the present disclosure, clustering the log record vectors in the vectorized log set to form different vector clusters includes:

[0020] According to a preset neighborhood parameter, determine the ∈-neighborhood of each of the log record vectors in the vectorized log set to obtain a core object set;

[0021] According to the first core object in the core object set, determining the log record vectors in the vectorized log set that are densely accessible to the first core object, to form a first vector cluster;

[0022] According to the second core object in the above-mentioned core object set, the above-mentioned log record vectors that are density-reachable by the above-mentioned second core object in the above-mentioned updated vectorized log set are determined to form a second vector cluster to obtain the above-mentioned different vector clusters, wherein the above-mentioned updated vectorized log set includes removing the above-mentioned log record vectors in the above-mentioned first vector cluster.

[0023] According to an embodiment of the present disclosure, the log records in the same log cluster are identified to obtain named entities including:

[0024] The initial model is trained using historical log data to obtain a named entity recognition model for identifying the above log records;

[0025] The log records in each of the log clusters are input into the named entity recognition model to output a named entity.

[0026] According to an embodiment of the present disclosure, the aforementioned use of historical log data to train the initial model to obtain a named entity recognition model for identifying the aforementioned log records includes:

[0027] Identify the operational entities in the field of IT operations to form a log entity dictionary;

[0028] Annotating the historical log data using the log entity dictionary to form an annotation set;

[0029] Construct word features and word boundary features to form a feature set;

[0030] Input the above annotation set and the above feature set into the target feature template to output a test data set;

[0031] The test data set is input into the initial model, and the initial model is trained to obtain the named entity recognition model.

[0032] According to an embodiment of the present disclosure, before the vectorization processing is performed on each of the above log records, the following steps are also included:

[0033] The fields with the preset format are marked by using rule definition or regular expression to shield the fields with the preset format.

[0034] According to an embodiment of the present disclosure, the above-mentioned fields with a preset format include time, special characters, universal resource identifiers, Internet protocols, curly brackets, square brackets, round brackets, underscores, slashes, and backslashes.

[0035] A second aspect of the present disclosure provides a log data processing device, comprising:

[0036] An acquisition module, used to acquire log data, wherein the log data includes at least one log record;

[0037] A vectorization module, used for performing vectorization processing on each of the above log records to obtain a vectorized log set containing log record vectors;

[0038] A clustering module, used for clustering the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records;

[0039] A named entity recognition module, used to recognize the log records in the same log cluster to obtain named entities; and

[0040] A generating module is used to generate a log template of the log cluster according to the named entity, wherein the log template is used to characterize the log structure characteristics of the log cluster.

[0041] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the log data processing method.

[0042] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned log data processing method.

[0043] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned log data processing method when executed by a processor.

[0044] According to an embodiment of the present disclosure, log data containing at least one log record is obtained, and the log record is vectorized to obtain a vectorized log set containing a log record vector; then the log record vector is clustered by clustering to form a log cluster containing similar log records; then the log records in each log cluster are identified to obtain a named entity, and a log template for characterizing the log structure characteristics of the log cluster is generated according to the named entity. The above method can effectively structure a large number of logs, remove a large amount of redundant data, and realize intelligent analysis of log data by clustering log records into different log clusters and extracting the log template of each log cluster, thereby reducing manual intervention, improving log analysis efficiency and operation and maintenance efficiency, providing a more effective analysis means for subsequent anomaly detection and automatic alarm, and better ensuring the stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0046] Figure 1 The application scenario diagram of the log data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0047] Figure 2 A flow chart schematically shows a log data processing method according to an embodiment of the present disclosure;

[0048] Figure 3 A flowchart for vectorizing log records is schematically shown;

[0049] Figure 4 A flow chart of a clustering processing method is schematically shown;

[0050] Figure 5 A flowchart of a method for determining a named entity recognition model for identifying log records is schematically shown;

[0051] Figure 6 The structure block diagram of the log data processing device according to the embodiment of the present disclosure is schematically shown;

[0052] Figure 7 A structural block diagram schematically shows a log data processing device according to another embodiment of the present disclosure; and

[0053] Figure 8 A block diagram of an electronic device suitable for implementing a log data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0055] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0056] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0057] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0058] Logs are data generated by information systems, and have the following characteristics: First, the sources are very wide, including logs and alarms generated by various devices and systems in traditional information system environments, as well as a large number of logs generated by mobile clients and sensors; second, the number is large, various machines and systems generate data all the time, and the total number is large; third, there are many types and non-uniform formats, and various systems generate different formats and meanings, and have different professional and technical requirements for technicians; fourth, the storage is dispersed, and different logs are stored in different systems and devices, requiring different methods to collect and read.

[0059] Log data reflects the real situation of the information system. Analyzing log data is an important basis for information operation and maintenance managers and security managers to troubleshoot and analyze attack sources in daily operation and maintenance. In related technologies, the method of analyzing log data is mainly to classify and identify log data, set up classifications according to formats, match corresponding data field processing paradigm rules, extract fields and assign values ​​to specified fields to complete data extraction.

[0060] However, the solutions in the related art need to predefine the log format and adapt each log format. If the log format changes, it is easy to cause format errors in the recognition of new logs, and the entire log recognition fails. It is necessary to establish new data format matching rules to solve the problem. Not only is the adaptability relatively poor, but the entire process usually requires technical support, which is time-consuming and labor-intensive, has a long cycle, and the user experience is relatively poor.

[0061] In response to the above technical problems, the present disclosure provides a log data processing method, including: obtaining log data, wherein the log data includes at least one log record; performing vectorization processing on each log record to obtain a vectorized log set containing a log record vector; performing clustering processing on the log record vectors in the vectorized log set to form different log clusters, wherein similar log records are contained in the same log cluster; identifying the log records in the same log cluster to obtain named entities; and generating a log template for the log cluster based on the named entity, wherein the log template is used to characterize the log structure characteristics of the log cluster.

[0062] It should be noted that the method and device for processing log data of the embodiments of the present disclosure can be used in the financial field and the computer technology field, and can also be used in any technical field other than the financial field and the computer technology field. The application field of the method and device for processing log data of the present disclosure is not limited.

[0063] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0064] Figure 1 The application scenario diagram of the log data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0065] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a network, a terminal device and a server. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0066] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0067] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0068] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0069] It should be noted that the log data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the log data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The log data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the log data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0070] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0071] The following will be based on Figure 1 The scene described by Figure 2 to Figure 5 The log data processing method of the disclosed embodiment is described in detail.

[0072] Figure 2 The flowchart of the log data processing method according to the embodiment of the present disclosure is schematically shown.

[0073] like Figure 2 As shown, the log data processing method of this embodiment includes operations S210 to S250, and the log data processing method can be executed by a terminal device or a server.

[0074] In operation S210 , log data is acquired, where the log data includes at least one log record.

[0075] In operation S220 , vectorization processing is performed on each log record to obtain a vectorized log set including log record vectors.

[0076] According to an embodiment of the present disclosure, the vectorization process may adopt, for example, the Word2 Voc word vector training model or the BERT (Bidirectional Encoder Representation from Transformers) word vector training model. The Word2Voc word vector training model is a static vector representation that does not consider the semantic word vector representation. Each word corresponds to only one word vector representation, and the word vector obtained for the same word is unchanged. The BERT word vector training model is a dynamic word vector representation that considers the semantic word vector representation. The same word obtains different vectorized representations depending on the context information.

[0077] In operation S230 , clustering is performed on the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records.

[0078] According to the embodiments of the present disclosure, log records with high similarity are aggregated into the same log cluster through clustering processing, which not only allows an overview of the log data, but also facilitates problem location and anomaly detection.

[0079] In operation S240 , log records in the same log cluster are identified to obtain named entities.

[0080] According to an embodiment of the present disclosure, when identifying log records in the same log cluster, for example, part of the log records may be selected for identification, or all of the log records may be selected for identification.

[0081] According to an embodiment of the present disclosure, identifying log records in the same log cluster may include, for example, identifying named entities in the log records in the same log cluster, that is, identifying named entities corresponding to the log records.

[0082] The specific process of identifying the named entities corresponding to log records includes: based on domain knowledge, determining the named entities of the logs in the domain from the operation and maintenance production practice, and forming a log entity dictionary with the named entities; then using the named entities in the log entity dictionary to annotate the historical log data; then using the historical log data annotated with the named entities to train the initial model to obtain a named entity recognition model; then inputting the log records that need to be recognized by the named entities into the trained named entity recognition model, and outputting the named entities corresponding to the log records, thereby realizing the recognition of the log records.

[0083] According to an embodiment of the present disclosure, the named entity may include, for example, date, Internet Protocol address, port (a type of interface), URL (Uniform Resource Locator), level, component, connection_pid (pid is a process identifier), etc.

[0084] According to an embodiment of the present disclosure, a URL refers to an identification method for completely describing the address of a web page or other resources on the Internet.

[0085] According to an embodiment of the present disclosure, the named entity may be accumulated from generation practice based on log domain knowledge.

[0086] According to the embodiments of the present disclosure, named entities generally refer to entities with specific meanings or strong referentiality in the text, usually including names of people, places, organizations, dates and times, proper nouns, etc. Named entity recognition is to extract named entities from unstructured input text, and more categories of named entities can be identified according to business needs. For named entities with strong regularity, such as percentages, URLs, E-mails, etc., they can be processed through regular expressions, and the unmatched fragments are handed over to the model for log entity information extraction.

[0087] Named entity recognition is a sequence labeling problem. Sequence labeling means finding the label corresponding to each element in a sequence given a sequence. For example, the input is a word sequence s corresponding to a log record. <word1,word2,......,word n >, the output is the entity sequence corresponding to the log record <entity1> <entity2>...... <entity n >.

[0088] In operation S250 , a log template of the log cluster is generated according to the named entity, wherein the log template is used to characterize a log structure feature of the log cluster.

[0089] According to an embodiment of the present disclosure, clustering processing is used to cluster log records into different log clusters, and a named entity recognition method is used to obtain named entities from these log clusters to generate a log template for each log cluster. The log template can more effectively structure the log data, remove a large amount of redundant data, and improve log analysis efficiency and operation and maintenance efficiency.

[0090] Figure 3 The flowchart of vectorizing log records is schematically shown.

[0091] like Figure 3 As shown, the vectorization process includes operations S310 to S340.

[0092] In operation S310, each log record is pre-processed.

[0093] According to an embodiment of the present disclosure, preprocessing of each log record may include, for example, stem extraction, lemma restoration, case conversion, and the like.

[0094] In operation S320, the pre-processed log records are segmented and stop words are removed to obtain a word corpus.

[0095] According to an embodiment of the present disclosure, before performing word segmentation and stop word removal processing, the process further includes constructing a stop word list in the field of log data.

[0096] According to an embodiment of the present disclosure, for example, word segmentation tools such as jieba and ltp are used to perform word segmentation and stop word removal on the preprocessed log records.

[0097] In operation S330, the word corpus is input into the vectorization model, the words are vectorized, and a word vector library is output; wherein the word vector library includes the word vector corresponding to each word.

[0098] According to an embodiment of the present disclosure, the vectorization model may include, for example, a Word2 Vec model. Words in the word prediction library are input into the Word2 Vec model to vectorize the words, and the word vector corresponding to each word is output.

[0099] In operation S340 , a log record vector corresponding to each log record is determined according to the words corresponding to each log record and the word vectors corresponding to the words, so as to obtain a vectorized log set.

[0100] According to an embodiment of the present disclosure, the log record vector can be obtained by calculating the vector sum of the word vectors of the words corresponding to each log record. For example, the word vectors of the words corresponding to log record X are divided into a, b, c, and d, and the log record vector corresponding to log record X is the vector sum of a, b, c, and d.

[0101] According to an embodiment of the present disclosure, clustering is performed on log record vectors in a vectorized log set to form different log clusters, including:

[0102] Clustering the log record vectors in the vectorized log set to form different vector clusters;

[0103] According to the log record vectors in the same vector cluster, the log record corresponding to each log record vector is determined to form a log cluster corresponding to the vector cluster.

[0104] According to an embodiment of the present disclosure, for example, the vectorized log set obtained in operation S220 is D M =(M1, M2, M3, ..., Mn), where M1, M2, M3, ..., Mn correspond to the vectorized log set D M A log record vector in , n represents the total number of log record vectors. M Clustering processing, vectorizing the log set D M Divide into multiple vector clusters C1, C2, ..., C m , where m represents the total number of vector clusters obtained by clustering. The log records corresponding to the log record vectors in vector cluster C1 form the log cluster corresponding to vector cluster C1.

[0105] Figure 4 The flowchart of the clustering processing method is schematically shown.

[0106] like Figure 4 As shown, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) log clustering method is used for processing, and the method includes operations S410 to S430.

[0107] In operation S410, according to a preset neighborhood parameter, an ∈-neighborhood of each log record vector in the vectorized log set is determined to obtain a core object set.

[0108] According to an embodiment of the present disclosure, for example, the vectorized log set is a sample set D = {x1, x2, ..., x m }, x1, x2, ..., x m is a log record vector, and the domain parameter is (∈, MinPts). j , x j ∈D, whose ∈-neighborhood contains all the nodes in D that are related to x j The distance between x and ∈ is not greater than that of the sample, if x j The ∈-neighborhood of contains at least MinPts samples, then x j is a core object. According to the above method, all the core objects in the sample set D are determined to form a core object set.

[0109] In operation S420 , based on a first core object in a core object set, a log record vector that is densely reachable by the first core object in a vectorized log set is determined to form a first vector cluster.

[0110] According to an embodiment of the present disclosure, for example, for a core object x i , if x j Located at x i ∈-neighborhood, then x is called j By x i Density direct; for x i With x j , if there exists a sample sequence p1, p2, ..., p n , where p1 = x i , p n =x j And p i+1 By p i The density is directly proportional to x j By x i Density can reach.

[0111] According to an embodiment of the present disclosure, according to the core object x i , determine the core object x i Density-reachable log record vectors,form the first vector cluster.

[0112] In operation S430, based on the second core object in the core object set, a log record vector that is density-reachable by the second core object in the updated vectorized log set is determined to form a second vector cluster to obtain a different vector cluster, wherein the updated vectorized log set includes removing the log record vector in the first vector cluster.

[0113] According to an embodiment of the present disclosure, after the log record vectors in the first vector cluster are removed from the vectorized log set, the log record vectors with density reachable by the second core object are determined according to the second core object to form a second vector cluster.

[0114] According to an embodiment of the present disclosure, each time a vector cluster is formed, the log record vectors in the vector cluster are removed from the vectorized log set, and operation S420 is repeatedly performed according to the vectorized log set after each update until all core objects are traversed or removed.

[0115] According to the embodiments of the present disclosure, the DBSCAN method can be used for clustering dense data sets of any shape, and the clustering results are unbiased. Clustering calculations can be performed one by one without serious degradation of the calculation speed, and the clustering effect is good.

[0116] According to an embodiment of the present disclosure, log records in the same log cluster are identified, and the obtained named entities include:

[0117] The initial model is trained using historical log data to obtain a named entity recognition model for identifying log records;

[0118] The log records in each log cluster are input into the named entity recognition model and the named entities are output.

[0119] Figure 5 A flowchart of a method for determining a named entity recognition model for identifying log records is schematically shown.

[0120] like Figure 5 As shown, the method for determining a named entity recognition model for identifying log records includes operations S510 to S550.

[0121] In operation S510, operation and maintenance entities in the field of information technology operation and maintenance are determined to form a log entity dictionary.

[0122] According to an embodiment of the present disclosure, the operation and maintenance entity may include, for example, date, Internet Protocol address, port interface, same resource locator, level, component, process identifier, name, place name, time, etc.

[0123] In operation S520 , the historical log data is annotated using a log entity dictionary to form an annotation set.

[0124] According to an embodiment of the present disclosure, the operation and maintenance entities in the log entity dictionary are used to annotate the historical log data to form an annotation set.

[0125] In operation S530 , word features and term boundary features are constructed to form a feature set.

[0126] According to an embodiment of the present disclosure, word features may include, for example, part of speech, whether it is a feature word, whether it is a place, whether it is the end of a sentence, etc. Word features include, for example, word-level features, dictionary features, and document-level features. Word-level features include whether it ends with a period, whether it contains numbers, part of speech, word n-gram (a language model), etc. Dictionary features rely on external dictionary definitions, such as predefined vocabularies, etc. Document-level features are calculated based on the entire corpus document set, such as word frequency, co-occurring words, etc. in the document set. Part-of-speech features refer to nouns, verbs, etc. Word boundary features are used to mark the position of words, using the BIO marking method, where B represents the starting position of the entity, I represents the middle or end position of the entity, and O represents that the corresponding character is not an entity.

[0127] In operation S540 , the annotation set and the feature set are input into a target feature template to output a test data set.

[0128] According to an embodiment of the present disclosure, the target feature template may include, for example, a Unigram template and a Bigram template. The Unigram template generates a state feature function sl(yi, x, i) of a CRF (Conditional Random Field), where yi is a label, x is an observation sequence, and i is the current node position. The Bigram template generates a transfer feature function tk(yi, yi-1, x, i), where yi is a label, x is an observation sequence, and i is the current node position.

[0129] According to an embodiment of the present disclosure, the annotation set and the feature set input into the target feature template are converted into a unified expression format to facilitate algorithm processing of the initial model.

[0130] In operation S550, the test data set is input into the initial model, and the initial model is trained to obtain a named entity recognition model.

[0131] According to an embodiment of the present disclosure, the initial model may include a CRF model, for example.

[0132] According to an embodiment of the present disclosure, the test data set may include a training set and a test set, wherein the training set is used to train the initial model, and the test set is used to test the accuracy of the trained named entity recognition model. The ratio of the training set to the test set may be 7:3.

[0133] According to an embodiment of the present disclosure, the named entity recognition model includes a loss function,

[0134]

[0135] Here, D represents the training sample (m sequences): D = [(x1, y1), (x2, y2), ..., (x m ,y m )], after taking partial derivative of λ, conventional optimization methods, such as gradient descent iterative optimization of model parameters, are used to predict the output sequence given the input sequence, that is, to find the optimal sequence that maximizes the objective function. This is a dynamic programming problem and can be decoded using the Viterbi algorithm.

[0136] According to an embodiment of the present disclosure, before vectorizing each log record, the following steps are further included:

[0137] The fields with the preset format are marked by using rule definition or regular expression to mask the fields with the preset format.

[0138] According to an embodiment of the present disclosure, shielding fields having a preset format helps to improve the accuracy of clustering.

[0139] According to an embodiment of the present disclosure, the fields with a preset format include time, special characters, universal resource identifiers, Internet protocols, curly brackets, square brackets, round brackets, underscores, slashes, and backslashes.

[0140] Based on the above log data processing method, the present disclosure also provides a log data processing device. Figure 6 The device is described in detail.

[0141] Figure 6 The structure block diagram of the log data processing device according to the embodiment of the present disclosure is schematically shown.

[0142] like Figure 6 As shown, the log data processing device 600 of this embodiment includes an acquisition module 610 , a vectorization module 620 , a clustering module 630 , a named entity recognition module 640 and a generation module 650 .

[0143] The acquisition module 610 is used to acquire log data, wherein the log data includes at least one log record. In one embodiment, the acquisition module 610 can be used to perform the operation S210 described above, which will not be described in detail here.

[0144] The vectorization module 620 is used to perform vectorization processing on each log record to obtain a vectorized log set including log record vectors. In one embodiment, the vectorization module 620 can be used to perform the operation S220 described above, which will not be described in detail here.

[0145] The clustering module 630 is used to perform clustering processing on the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records. In one embodiment, the clustering module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0146] The named entity recognition module 640 is used to recognize log records in the same log cluster to obtain named entities. In one embodiment, the named entity recognition module 640 can be used to perform the operation S240 described above, which will not be described in detail here.

[0147] The generation module 650 is used to generate a log template of a log cluster according to the named entity, wherein the log template is used to characterize the log structure characteristics of the log cluster. In one embodiment, the generation module 650 can be used to perform the operation S250 described above, which will not be repeated here.

[0148] According to an embodiment of the present disclosure, the vectorization module includes a preprocessing unit, a word segmentation and stop word removal unit, a vectorization unit, and a determination unit.

[0149] The preprocessing unit is used to preprocess each log record.

[0150] The word segmentation and stop word removal unit is used to perform word segmentation and stop word removal on the preprocessed log records to obtain a word corpus.

[0151] The vectorization unit is used to input the word corpus into the vectorization model, vectorize the words, and output a word vector library; wherein the word vector library includes the word vector corresponding to each word.

[0152] The first determination unit is used to determine the log record vector corresponding to each log record according to the words corresponding to each log record and the word vectors corresponding to the words, so as to obtain a vectorized log set.

[0153] According to an embodiment of the present disclosure, the clustering module includes a clustering unit and a determining unit.

[0154] The clustering unit is used to perform clustering processing on the log record vectors in the vectorized log set to form different vector clusters.

[0155] The second determining unit is used to determine the log record corresponding to each log record vector according to the log record vectors in the same vector cluster, so as to form a log cluster corresponding to the vector cluster.

[0156] According to an embodiment of the present disclosure, the clustering unit includes a first determining subunit, a second determining subunit, and a third determining subunit.

[0157] The first determining subunit is used to determine the ∈-neighborhood of each log record vector in the vectorized log set according to a preset neighborhood parameter to obtain a core object set.

[0158] The second determining subunit is used to determine, according to the first core object in the core object set, the log record vectors that are densely accessible to the first core object in the vectorized log set to form a first vector cluster.

[0159] The third determination subunit is used to determine the log record vectors that are densely reachable by the second core object in the updated vectorized log set according to the second core object in the core object set, to form a second vector cluster to obtain a different vector cluster, wherein the updated vectorized log set includes removing the log record vectors in the first vector cluster.

[0160] According to an embodiment of the present disclosure, the named entity recognition module includes a model training unit and an input unit:

[0161] The model training unit is used to train the initial model using historical log data to obtain a named entity recognition model for identifying log records.

[0162] The input unit is used to input the log records in each log cluster into the named entity recognition model and output the named entity.

[0163] According to an embodiment of the present disclosure, the model training unit includes a fourth determination subunit, a labeling subunit, a construction subunit, an input subunit, and a model training subunit.

[0164] The fourth determination subunit is used to determine the operation and maintenance entities in the field of information technology operation and maintenance to form a log entity dictionary.

[0165] The annotation subunit is used to annotate the historical log data using the log entity dictionary to form an annotation set.

[0166] Construct subunits for constructing word features and word boundary features to form a feature set.

[0167] The input subunit is used to input the annotation set and feature set into the target feature template to output the test data set.

[0168] The model training subunit is used to input the test data set into the initial model, train the initial model, and obtain a named entity recognition model.

[0169] Figure 7 The structure block diagram of a log data processing device according to another embodiment of the present disclosure is schematically shown.

[0170] like Figure 7 As shown, the log data processing device 600 of this embodiment includes not only the above-mentioned acquisition module 610, vectorization module 620, clustering module 630, named entity recognition module 640, and generation module 650, but also a marking module 660.

[0171] The marking module 660 is used to mark the fields with the preset format by using rule definition or regular expression to shield the fields with the preset format.

[0172] According to an embodiment of the present disclosure, the fields with a preset format include time, special characters, universal resource identifiers, Internet protocols, curly brackets, square brackets, round brackets, underscores, slashes, and backslashes.

[0173] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 610, the vectorization module 620, the clustering module 630, the named entity recognition module 640, the generation module 650 and the marking module 660 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the vectorization module 620, the clustering module 630, the named entity recognition module 640, the generation module 650 and the marking module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 610, the vectorization module 620, the clustering module 630, the named entity recognition module 640, the generation module 650 and the labeling module 660 can be at least partially implemented as a computer program module, which can perform the corresponding function when it is executed.

[0174] Figure 8 A block diagram of an electronic device suitable for implementing a log data processing method according to an embodiment of the present disclosure is schematically shown.

[0175] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0176] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0177] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

[0178] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0179] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0180] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0181] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0182] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0183] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0184] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0185] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0186] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0187] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure. < / entity1>

Claims

1. A log data processing method, comprising: Acquire log data, wherein the log data includes at least one log record; Performing vectorization processing on each of the log records to obtain a vectorized log set including log record vectors; performing clustering processing on the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records; Identifying the log records in the same log cluster to obtain named entities; and Generating a log template of the log cluster according to the named entity, wherein the log template is used to characterize a log structure feature of the log cluster; The vectorizing process of each log record to obtain a vectorized log set including log record vectors includes: Preprocessing each of the log records; Performing word segmentation and stop word removal processing on the preprocessed log records to obtain a word corpus; Input the word corpus into the vectorization model, vectorize the words, and output a word vector library; wherein the word vector library includes the word vector corresponding to each of the words; Determine, according to the word corresponding to each log record and the word vector corresponding to the word, a log record vector corresponding to each log record to obtain the vectorized log set; The clustering of the log record vectors in the vectorized log set to form different log clusters includes: Performing clustering processing on the log record vectors in the vectorized log set to form different vector clusters; According to the log record vectors in the same vector cluster, determining the log record corresponding to each log record vector to form the log cluster corresponding to the vector cluster; The clustering process of the log record vectors in the vectorized log set to form different vector clusters includes: According to a preset neighborhood parameter, determining the ∈-neighborhood of each of the log record vectors in the vectorized log set to obtain a core object set; According to a first core object in the core object set, determining the log record vectors in the vectorized log set that are densely reachable by the first core object to form a first vector cluster; According to the second core object in the core object set, the log record vectors that are density-reachable by the second core object in the updated vectorized log set are determined to form a second vector cluster to obtain a different vector cluster, wherein the updated vectorized log set includes removing the log record vectors in the first vector cluster.

2. According to the method of claim 1, the step of identifying the log records in the same log cluster to obtain the named entity comprises: Using historical log data to train an initial model to obtain a named entity recognition model for identifying the log records; The log records in each of the log clusters are input into the named entity recognition model, and a named entity is output.

3. According to the method of claim 2, the step of training the initial model using historical log data to obtain a named entity recognition model for identifying the log record comprises: Identify the operational entities in the field of IT operations to form a log entity dictionary; Annotating the historical log data using the log entity dictionary to form an annotation set; Construct word features and word boundary features to form a feature set; Inputting the annotation set and the feature set into a target feature template to output a test data set; The test data set is input into the initial model, and the initial model is trained to obtain the named entity recognition model.

4. The method according to claim 1, further comprising, before vectorizing each of the log records: The fields with the preset format are marked by using rule definition or regular expression to shield the fields with the preset format.

5. The method according to claim 4, wherein: The fields with the preset format include time, special characters, universal resource identifiers, Internet protocols, curly brackets, square brackets, round brackets, underscores, slashes, and backslashes.

6. A log data processing device, comprising: An acquisition module, used to acquire log data, wherein the log data includes at least one log record; A vectorization module, used for performing vectorization processing on each of the log records to obtain a vectorized log set containing log record vectors; A clustering module, used for clustering the log record vectors in the vectorized log set to form different log clusters, wherein the same log cluster contains similar log records; a named entity recognition module, used for identifying the log records in the same log cluster to obtain named entities; and A generating module, used for generating a log template of the log cluster according to the named entity, wherein the log template is used for characterizing the log structure characteristics of the log cluster; The vectorization module comprises: A preprocessing unit, used for preprocessing each of the log records; A word segmentation and stop word removal unit, used for performing word segmentation and stop word removal on the pre-processed log records to obtain a word corpus; A vectorization unit, used for inputting the word corpus into a vectorization model, vectorizing the words, and outputting a word vector library; wherein the word vector library includes a word vector corresponding to each of the words; A first determining unit, configured to determine a log record vector corresponding to each log record according to the word corresponding to each log record and the word vector corresponding to the word, so as to obtain the vectorized log set; The clustering module comprises: A clustering unit, used for clustering the log record vectors in the vectorized log set to form different vector clusters; A second determining unit, configured to determine the log record corresponding to each log record vector according to the log record vectors in the same vector cluster, so as to form the log cluster corresponding to the vector cluster; The clustering unit comprises: A first determining subunit, configured to determine the ∈-neighborhood of each of the log record vectors in the vectorized log set according to a preset neighborhood parameter, so as to obtain a core object set; A second determining subunit is configured to determine, according to a first core object in the core object set, the log record vectors in the vectorized log set that are densely reachable by the first core object, to form a first vector cluster; The third determination subunit is used to determine the log record vector that is densely reachable by the second core object in the core object set in the updated vectorized log set, based on the second core object in the core object set, to form a second vector cluster to obtain a different vector cluster, wherein the updated vectorized log set includes removing the log record vector in the first vector cluster.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images