Method and device for processing log and electronic equipment
Through log analysis tools and natural language processing technology of word segmentation, the problem of low log processing efficiency and accuracy in the existing technology is solved, and more efficient log analysis and cluster analysis are achieved, reducing the need for manual configuration.
Patent Information
- Application Number
- CN202411109172.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-15
AI Technical Summary
When faced with large-scale, complex and dynamically changing log information, existing log processing methods have poor efficiency and accuracy, making it difficult to process natural language descriptions in the log, and require a lot of manual configuration, which affects processing efficiency.
The log analysis tool performs log parsing operations, converts the pending log data into the target data in a set format, and uses the lexicon to perform natural language processing, perform clustering analysis, reduces manual configuration requirements, and improves processing efficiency and accuracy.
It improves log processing efficiency, can better understand and process natural language information, realize more accurate information extraction and log clustering, reduces human errors, and improves the processing ability of log data.
Smart Images

Figure CN118964616A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of log processing, for example, to a method and device for processing logs, and an electronic device. Background Art
[0002] At present, with the rapid development of information technology, systems and applications continue to generate a large amount of log data. Log files usually record a variety of data such as system operation status, error information, user behavior, etc. The value of log data is becoming increasingly prominent, especially in the fields of IT (Information Technology) operation and maintenance, security analysis and business intelligence, which plays an important role in system monitoring, security auditing, performance optimization, etc. However, due to the unstructured nature of log data and the massive amount of information, how to efficiently and accurately process and analyze this data has always been a challenge in the field of IT management and operation and maintenance. Traditional log processing methods mainly rely on simple keyword searches and regular expression matching. This method is more effective when processing small-scale and simple-structured log data, but its efficiency and accuracy are poor when facing large-scale, complex and dynamically changing log information. In addition, traditional methods have difficulty processing natural language descriptions in logs and cannot deeply understand and analyze the meaning and context of log content.
[0003] In the related art, a log analysis tool based on regular expressions is disclosed, including Logstash and Graylog, which can collect, filter and store logs. Regular expressions are used for pattern matching to identify and extract specific information.
[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:
[0005] Log analysis tools require a lot of manual configuration and have difficulty dealing with log data with changing formats. Therefore, log processing efficiency is affected during the log data processing process.
[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.
[0008] The embodiments of the present disclosure provide a method and device for processing logs, and an electronic device to improve log processing efficiency.
[0009] In some embodiments, the method for processing logs includes: obtaining log data to be processed; based on a log analysis tool, performing a log parsing operation to convert the log data to be processed into target data in a set format; based on a word segmentation library, performing natural language processing on the target data to obtain natural language data; performing cluster analysis on the target data based on the natural language data to obtain cluster data after clustering grouping is completed.
[0010] Optionally, obtaining the log data to be processed includes: obtaining original log data; performing time matching on the original log data to merge multiple log lines; and preprocessing the merged original log data to obtain the log data to be processed.
[0011] Optionally, performing time matching on the original log data includes: determining a format of a timestamp in the original log data; writing a target regularized expression according to the format of the timestamp; and performing time matching on the original log data using the target regularized expression.
[0012] Optionally, a log parsing operation is performed to convert the log data to be processed into target data in a set format, including: initializing a log parser; based on the log parser, using a regular expression to match and parse the log data to be processed to obtain parsed data; and converting the target log data to be processed that matches the parsed data into target data in a set format.
[0013] Optionally, initializing the log parser includes: calling a constructor; initializing parameters of the log parser according to the constructor; the parameters of the log parser include log parsing depth, log parsing score, maximum length of each log line, and log parsing regular expression.
[0014] Optionally, the target log data to be processed that matches the parsed data is converted into target data in a set format, including: calling a conversion function; based on the conversion function, using a regular expression to traverse and search each line of data in the target log data to be processed, and matching log entries; when the current line matches successfully, extracting the value of each field from the matching object to form a log message, and storing it in a target list; using the log message in the target list to create the target data in a set format.
[0015] Optionally, natural language processing is performed on the target data, including: performing word segmentation processing on the target data to obtain word segmentation results; analyzing the word segmentation results based on a preset natural language processing model to obtain keyword information and perform semantic understanding of the keyword information.
[0016] Optionally, cluster analysis is performed on the target data according to the natural language data, including: constructing a search tree according to keywords of the natural language data; and clustering and grouping the target data according to the search tree.
[0017] In some embodiments, the apparatus for processing logs includes a processor and a memory storing program instructions, and the processor is configured to execute the method for processing logs as described above when running the program instructions.
[0018] In some embodiments, the electronic device includes: an electronic device body; and the device for processing logs as described above, installed in the electronic device body.
[0019] The method, device, and electronic device for processing logs provided by the embodiments of the present disclosure can achieve the following technical effects:
[0020] In the disclosed embodiment, after obtaining the log data to be processed, the log parsing operation is performed through the log analysis tool, which can reduce the need for manual configuration, improve the processing speed under large amounts of data, and improve the response speed of the log parsing operation, thereby improving the log processing efficiency. Then, the target data is subjected to natural language processing through the word segmentation library, so that the natural language information in the log can be better understood and processed. By performing natural language understanding on the log content, information can be extracted more accurately, effective information can be quickly obtained, and the log processing efficiency can be further improved. In addition, after performing natural language processing on the target data and obtaining the natural language data, a suitable clustering algorithm is used to perform cluster analysis on the target data based on the natural language data, so that more accurate and relevant log clustering can be achieved, and the ability to process log data can be improved.
[0021] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:
[0023] Figure 1 is a schematic diagram of a log natural language analysis and processing system provided by an embodiment of the present disclosure;
[0024] Figure 2 is a schematic diagram of another log natural language analysis and processing system provided by an embodiment of the present disclosure;
[0025] Figure 3 is a schematic diagram of a method for processing logs provided by an embodiment of the present disclosure;
[0026] Figure 4 is a schematic diagram of a natural language processing model provided by an embodiment of the present disclosure;
[0027] Figure 5 is a schematic diagram of another method for processing logs provided by an embodiment of the present disclosure;
[0028] Figure 6 It is a schematic diagram of a device for processing logs provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0029] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.
[0030] The terms "first", "second", etc. in the technical solutions described in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the disclosed embodiments described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.
[0031] Unless otherwise stated, the term "plurality" means two or more.
[0032] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.
[0033] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.
[0034] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.
[0035] Combination Figure 1As shown, the embodiment of the present disclosure provides a log natural language analysis and processing system 10, including an application service module 20 and a log clustering service module 30. The application service module 20 is connected to the log clustering service module 30. The application service module 10 is used to generate original log data and send the original log data to the log clustering service module 30. The original log data records various operations and event information generated by the application or server during operation. The log clustering service module 30 can be set based on a log analysis tool to perform natural language analysis on the original log data.
[0036] Optionally, the log natural language analysis and processing system 10 further includes: a device 300 for processing logs. The device 300 for processing logs includes a processor. The device 300 for processing logs is connected to the log clustering service module 30, and the processor can execute the method for processing logs by calling the log clustering service module 30.
[0037] Combination Figure 2 As shown, optionally, the log clustering service module 30 includes: a log parsing extension module 31, a log parsing service module 32 and a log processing module 33. The log parsing extension module 31 is connected to the log parsing service module 32 and the log processing module 33 respectively. The log parsing extension module 31 is used to execute the _init_ method to perform initialization settings, and execute the pares method based on the log parser to perform natural language analysis processing of the log. The log parsing service module 32 is used to initialize the log parser. The log processing module 33 is used to obtain the original log data and perform preprocessing to obtain the log data to be processed.
[0038] Combination Figure 3 As shown, the present disclosure is applied to the above-mentioned log natural language analysis and processing system. A method for processing logs is provided in an embodiment of the present disclosure. The execution subject of the method may be a processor. The method includes:
[0039] S001: The processor obtains log data to be processed.
[0040] S002: The processor performs a log parsing operation based on a log analysis tool to convert the log data to be processed into target data in a set format.
[0041] S003, the processor performs natural language processing on the target data based on the word segmentation library to obtain natural language data.
[0042] S004, the processor performs cluster analysis on the target data according to the natural language data to obtain cluster data after clustering and grouping.
[0043] In the disclosed embodiments, after obtaining the log data to be processed, the log parsing operation is performed through the log analysis tool, which can reduce the need for manual configuration, improve the processing speed under large amounts of data, and improve the response speed of the log parsing operation, thereby improving the log processing efficiency. By performing natural language processing on the target data through the word segmentation library, the natural language information in the log can be better understood and processed. By performing language understanding on the log content, information can be extracted more accurately, effective information can be quickly obtained, and the log processing efficiency can be further improved. In addition, after performing natural language processing on the target data and obtaining the natural language data, a suitable clustering algorithm is used to perform cluster analysis on the target data based on the natural language data, which can achieve more accurate and relevant log clustering and improve the ability to process log data.
[0044] Optionally, the log analysis tool includes a LogParser tool.
[0045] The LogParser tool is a log parsing tool that provides rich functions and flexible parsing mechanisms. It can analyze text-based log files, XML files, CSV (comma separated value) files, as well as operating system event logs, registry, file system, Active Directory and other data sources. It is suitable for various log analysis needs.
[0046] Optionally, the word segmentation library includes the jieba word segmentation library.
[0047] Jieba is a popular Chinese text segmentation Python library. It supports three segmentation modes: precise mode, full mode and search engine mode, and supports traditional Chinese segmentation and custom dictionaries. Jieba is a simple, efficient and flexible Chinese word segmentation tool.
[0048] Optionally, obtaining the log data to be processed includes: obtaining original log data; performing time matching on the original log data to merge multiple log lines; and preprocessing the merged original log data to obtain the log data to be processed.
[0049] In an application or server, raw log data can be collected. The raw log data records various operations and event information generated during the operation of the application or server. For log files that scatter a single event across multiple lines, regular expressions can be used to time-match the raw log data, thereby merging multiple lines of logs. The merged raw log data is preprocessed, where the preprocessing operation includes one or more of data cleaning, data formatting, and data validation. Finally, the log data to be processed can be stored in the log server, and the relevant log records can be extracted from the stored log data to be processed for further parsing and analysis based on the specific problems or requirements that need to be analyzed.
[0050] Optionally, performing time matching on the original log data includes: determining a format of a timestamp in the original log data; writing a target regularized expression according to the format of the timestamp; and performing time matching on the original log data using the target regularized expression.
[0051] In the log system, the original log data may be displayed in multiple lines, each line starting with a timestamp. When performing time matching on the original log data, you need to first determine the format of the timestamp in the original log data, and write a target regular expression based on the format of the timestamp. Then read the original log data line by line, and use the target regular expression to check whether each line of the original log data starts with a timestamp. If so, merge the current line into the previous line or create a new log entry as needed to achieve time matching of the original log data.
[0052] Optionally, a log parsing operation is performed to convert the log data to be processed into target data in a set format, including: initializing a log parser; based on the log parser, using a regular expression to match and parse the log data to be processed to obtain parsed data; and converting the target log data to be processed that matches the parsed data into target data in a set format.
[0053] In this embodiment, based on the LogParser tool, the parse method is executed by the log parser to perform log parsing operations on the log data to be processed. In LogParser, the _init_ method is executed to initialize the log parser. In the log parser, a regular expression is used to match and parse the log data to be processed, and the parsed data is obtained, and the parsed data is stored in DataFrame format. Finally, the log_to_dataframe method is executed to convert the matched log data to be processed into DataFrame format for subsequent processing.
[0054] Optionally, initializing the log parser includes: calling a constructor; initializing parameters of the log parser according to the constructor; the parameters of the log parser include log parsing depth, log parsing score, maximum length of each log line, and log parsing regular expression.
[0055] In this embodiment, a class named MyLogParser can be defined by inheriting from a base class named LogParser. The constructor _init_ of the MyLogParser class is called to initialize an instance of the log parser. In the constructor, the defined parameters include: log_format, the format of the log, which is used to define the structure of the log; indir, the input directory, the storage location of the log file, the default is the current directory; outdir, the output directory, the parsed log data will be saved in the output directory; depth, the log parsing depth, which is related to the recursive depth when parsing the log; st, a score, which is related to the log parsing score; maxChild, the maximum number of sub-items for each log item; rex, a regular expression, used for log parsing; keep_para, a Boolean value indicating whether to retain certain parameters; max_len, the maximum length of each line of log. The instance attributes are initialized through self.logs and self.max_len. self.logs is used to store the log data to be parsed, and the initial value is None; self.max_len is used to set the maximum length of each line of log, which is used to truncate long log lines or limit the characters when processing logs. Finally, through the super()._init_ call, the above parameters are passed to the constructor of the base class LogParser to complete the initialization process of the base class.
[0056] Optionally, the target log data to be processed that matches the parsed data is converted into target data in a set format, including: calling a conversion function; based on the conversion function, using a regular expression to traverse and search each line of data in the target log data to be processed, and matching log entries; when the current line matches successfully, extracting the value of each field from the matching object to form a log message, and storing it in a target list; using the log message in the target list to create the target data in a set format.
[0057] In this embodiment, the target log data to be processed is loaded and converted into the target data of the set format through the load_data method. The set format includes the DataFrame format. In load_data, the generate_logformat_regex method is called, and self.log_format is used as a parameter to generate the regular expression regex corresponding to the log format and the column name headers for the DataFrame. Then the log_to_dataframe method is used to convert the log data self.logs, the regular expression regex, the column name headers and the log format self.log_format as parameters to convert the log data into the DataFrame format and assign it to self.df_log. In the log_to_dataframe method, the initialization of log_messages and linecount is first performed. Log_messages is used to store the parsed log information list, that is, the target list, and linecount is used to record the number of log lines that are successfully matched. Use the for loop to traverse each line of log in the logs list, and then try to match the regular expression. During the matching process, the try block attempts to match each line of data in the target log data to be processed using regex.search(line.strip()). If the match is successful, use match.group(header) to extract the value of each specified column, add these values to log_messages as a list message, and update the linecount counter. At the same time, use the except block to capture and handle possible exceptions. For example, when the regular expression does not match, print a warning message and skip the current line. Finally, use pd.DataFrame(log_messages, columns=headers) to create a DataFrame, where headers is the column name, use the insert method to insert a column named "LineId" at the beginning of the DataFrame, initialize it to None, and update the "LineId" column to an incremental number starting from 1. After all the target log data to be processed are matched, print the number of successfully converted log lines, and return the DataFrame logdf containing the parsed log data.
[0058] Optionally, natural language processing is performed on the target data, including: performing word segmentation processing on the target data to obtain word segmentation results; analyzing the word segmentation results based on a preset natural language processing model to obtain keyword information and perform semantic understanding of the keyword information.
[0059] In this embodiment, when natural language processing is performed on the target data, the target data also needs to be loaded in the processing environment. First, the print function is used to output the length of the log to be parsed, that is, the length of the logs list, and datetime.now() is used to obtain the current time, that is, the start time of the processing process, for possible performance evaluation later. The incoming logs are assigned to the instance variable self.logs to set the log data. Create an instance rootNode of the Node class to initialize the root node for subsequent clustering operations, and initialize an empty list logCluL as a log clustering list for storing log clustering results. Then call the self.load_data() method to load the target data and convert the target data into a DataFrame. The count variable is set to initialize the counter to record the number of log lines processed. Use a for loop combined with the iterrows() method to traverse each row in the self.df_log DataFrame, and for each row, get the row index idx and row data line. During the traversal process, extract the "LineId" and "Class" fields, use the strip() method to remove possible blank characters, and then call the self.preprocess() method to preprocess the "Content" field and cut it to the length specified by self.max_len. Use the jieba.cut method to perform Chinese word segmentation on the preprocessed log information logMsg, and store the result in logmessageL. Finally, analyze the word segmentation results based on the preset natural language processing model, obtain keyword information, and perform semantic understanding of keyword information to obtain natural language data.
[0060] Combination Figure 4As shown, an embodiment of the present disclosure provides a natural language processing model. After obtaining the vocabulary ROOT through jieba word segmentation, LogTemp is used to store the log information after word segmentation. Classify according to the length Length of each log information. Then, according to the specific data in the log information, such as the operation type and data volume of the log, the index number of the log information is recorded. Among them, the operation type includes the sending operation "Send Data" and the receiving operation "ReceivedData"; the data volume includes "1234bytes", which means that 1234bytes of data are sent or received. In cluster analysis, similar log entries may be grouped together to help identify common data transmission patterns or discover abnormal behavior. For example, after word segmentation, the log information obtained is: "Send Data 1234bytes" with a length of 4 or "Received Data from 192.168.1.1 1234bytes" with a length of 6. Then "Send Data 1234bytes" is classified as "Send Data*bytes", and its index number Logids is "1234", and "Received Data from 192.168.1.11234bytes" is classified as "Received Data from**bytes", its index number is "1234", and the IP address is "192.168.1.1 1234".
[0061] Optionally, cluster analysis is performed on the target data according to the natural language data, including: constructing a search tree according to keywords of the natural language data; and clustering and grouping the target data according to the search tree.
[0062] In this embodiment, a search tree is constructed based on the keywords of the natural language data. It is necessary to call the self.treeSearch() method, use the rootNode and the word segmentation result logmessageL to search, and try to match the existing log cluster. If matchCluster is None, it means that no matching cluster is found. A new Logcluster instance is created and added to the logCluL list and prefix tree. The target data is clustered and grouped according to the search tree. If a matching cluster is found, the cluster log template is updated and the log ID is added to the cluster.
[0063] Combination Figure 5 As shown, the embodiment of the present disclosure provides another method for processing logs, including:
[0064] S101, the application service module collects original log data.
[0065] S102, the application service module sends original log data to the log processing module.
[0066] S103, the log processing module performs time matching on the original log data to achieve multi-line log merging.
[0067] S104: The log processing module pre-processes the merged original log data to obtain log data to be processed.
[0068] S105, the log processing module sends the log data to be processed to the log parsing extension module.
[0069] S106, the log parsing extension module sends an initialization instruction to the log parsing service module.
[0070] S107, the log parsing service module initializes the log parser.
[0071] S108, the log parsing service module sends an initialization completion flag to the log parsing extension module.
[0072] S109, the log parsing extension module performs a log parsing operation based on the log parser to convert the log data to be processed into target data in a set format.
[0073] S110, the log parsing extension module performs natural language processing on the target data based on the word segmentation library to obtain natural language data.
[0074] S111, the log parsing extension module performs cluster analysis on the target data according to the natural language data to obtain cluster data after clustering and grouping.
[0075] The method for processing logs provided by the embodiment of the present disclosure is based on a log natural language analysis and processing system. The system can automatically parse a large amount of log data, which significantly improves work efficiency compared to manual processing. Using natural language processing technology, the system can intelligently identify the semantics and patterns in the log, reducing human errors. The system can automatically respond to specific log events according to preset rules, realize operation and maintenance automation, and reduce labor costs. Combined with LogParser and Jieba, the system can perform in-depth log analysis to mine key information and potential problems in the log. By analyzing historical log data, the system can predict future log trends and provide support for decision-making. The system can automatically detect abnormal patterns in logs and promptly discover potential security threats or system failures. Users can query logs through natural language without learning complex query languages, which reduces the threshold for use. The system provides visualization tools such as charts and reports to help users understand log data more intuitively. Users can customize log analysis rules according to their needs to make the system more in line with actual application scenarios. The system supports multiple log formats and can process different types of log data, increasing its applicability. Through natural language processing technology, the system can clean and preprocess log data to improve data quality. The system can integrate and analyze log data from different sources, providing a comprehensive data perspective. The modular design of the system makes it easy to add new features or expand existing features. The system can be integrated with other tools and platforms to build a complete log management solution. The system is suitable for cloud-native environments and can process logs generated by microservices and containers, adapting to the development trend of modern applications.
[0076] Therefore, the log natural language analysis and processing system based on LogParser and Jieba has the advantages of automated processing, powerful log analysis, user-friendliness, data processing flexibility and good scalability. These advantages give the system a significant advantage in the field of log management and provide strong support for the enterprise's IT operation and maintenance, security analysis and business intelligence. With the continuous advancement of technology, the system can continue to be optimized and expanded to provide users with more efficient and intelligent log analysis tools.
[0077] Combination Figure 6As shown, an embodiment of the present disclosure provides a device 300 for processing logs, including a processor 200 and a memory 201. Optionally, the device may also include a communication interface 202 and a bus 203. The processor 200, the communication interface 202, and the memory 201 may communicate with each other through the bus 203. The communication interface 202 may be used for information transmission. The processor 200 may call the logic instructions in the memory 201 to execute the method for processing logs of the above embodiment.
[0078] In addition, the logic instructions in the memory 201 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.
[0079] The memory 201 is a computer-readable storage medium that can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 200 executes the function application and data processing by running the program instructions / modules stored in the memory 201, that is, the method for processing logs in the above embodiment is implemented.
[0080] The memory 201 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 201 may include a high-speed random access memory and may also include a non-volatile memory.
[0081] An embodiment of the present disclosure provides an electronic device, comprising: an electronic device body, and the above-mentioned device for processing logs. The device for processing logs is installed in the electronic device body. The installation relationship described here is not limited to placement inside the electronic device, but also includes installation connections with other components of the electronic device, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the device for processing logs can be adapted to a feasible electronic device body, thereby realizing other feasible embodiments.
[0082] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for processing logs.
[0083] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0084] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the technical solutions recorded in this application. As used in the technical solutions recorded in this application, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0085] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0086] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0087] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A method for processing logs, It is characterized in that include: Get the log data to be processed; Based on the log analysis tool, perform log parsing operations to convert the log data to be processed into target data in a set format; Based on the word segmentation library, natural language processing is performed on the target data to obtain natural language data; Cluster analysis is performed on the target data according to the natural language data to obtain cluster data with completed clustering grouping.
2. The method for processing logs according to claim 1, It is characterized in that Get the log data to be processed, including: Get the original log data; Perform time matching on the original log data to merge multiple log lines; The merged original log data is preprocessed to obtain log data to be processed.
3. The method for processing logs according to claim 2, It is characterized in that Perform time matching on raw log data, including: Determine the format of timestamps in raw log data; Write the target regularization expression according to the format of the timestamp; Use the target regular expression to perform time matching on the raw log data.
4. The method for processing logs according to claim 1, It is characterized in that Perform log parsing operations to convert the log data to be processed into target data in a set format, including: Initialize the log parser; Based on the log parser, regular expressions are used to match and parse the log data to be processed to obtain parsed data; Convert the target log data to be processed that matches the parsed data into target data in a set format.
5. The method for processing logs according to claim 4, It is characterized in that Initialize the log parser, including: Call the constructor; Initialize the parameters of the log parser according to the constructor; the parameters of the log parser include log parsing depth, log parsing score, maximum length of each log line, and log parsing regular expression.
6. The method for processing logs according to claim 4, It is characterized in that Convert the target log data to be processed that matches the parsed data into target data in a set format, including: Call the conversion function; Based on the conversion function, use regular expressions to traverse and search each line of data in the target log data to be processed, and match the log entries; If the current row matches successfully, extract the value of each field from the matching object to form a log message and store it in the target list; Creates formatted target data using log messages from a target list.
7. The method for processing a log according to any one of claims 1 to 6, It is characterized in that Perform natural language processing on target data, including: Perform word segmentation on the target data to obtain the word segmentation result; The word segmentation results are analyzed based on the preset natural language processing model to obtain keyword information and perform semantic understanding of the keyword information.
8. The method for processing logs according to any one of claims 1 to 6, It is characterized in that Perform cluster analysis on target data based on natural language data, including: Build a search tree based on the keywords of natural language data; Cluster and group the target data based on the search tree.
9. A device for processing logs, comprising a processor and a memory storing program instructions, It is characterized in that The processor is configured to execute the method for processing logs according to any one of claims 1 to 8 when running the program instructions.
10. An electronic device, It is characterized in that include: Electronic device body; The device for processing logs according to claim 9 is installed in the electronic device body.
Citation Information
Patent Citations
Log analysis method and system and electronic equipment
CN111950263A
Log analysis method and log analysis device
CN112667672A
Log data processing method and device, equipment and storage medium
CN116701147A
Log-based train braking capacity influence analysis method and system and medium
CN117290407A
Log semantic vectorization and hierarchical clustering-based log analysis method
CN117520033A
