A method and system for automatically grouping logs based on text beginning and ending information

Abstract

The present invention discloses a log automatic grouping method and system based on text start and end information, and relates to the field of data processing. The method comprises: obtaining log text data to be processed; determining a start mark in the log text data to be processed; the start mark is used to identify a string or symbol at the beginning of each group of logs; determining an end mark in the log text data to be processed; the end mark is used to identify a string or symbol at the end of each group of logs; scanning the content of the log text data to be processed, and using a string matching algorithm to determine the positions of all start marks and all end marks in the log text data to be processed; and dividing the log text data to be processed into a plurality of log groups according to the positions of all start marks and all end marks in the log text data to be processed. The present invention improves the grouping efficiency and accuracy of log data.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and system for automatically grouping logs based on text beginning and ending information. Background Art

[0002] Log data processing is an essential part of many systems. However, traditional log processing methods have the following problems:

[0003] 1. A large number of logs lack structured identification and cannot be automatically classified and grouped. They need to be manually checked and analyzed line by line, which is inefficient.

[0004] 2. Only relying on fixed-pattern regular expressions for matching, when faced with log data with complex formats and various changes, the recognition accuracy is low and the grouping error rate is high.

[0005] 3. The inability to identify logs across rows often results in the same log being split and scattered into different groups, resulting in distorted results.

[0006] 4. Manual grouping and classification is very time-consuming, which seriously restricts the efficiency of subsequent log analysis and processing.

[0007] Based on the above problems, a new method is urgently needed to improve the efficiency of log data processing. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for automatically grouping logs based on text beginning and ending information, which can improve the grouping efficiency and accuracy of log data.

[0009] To achieve the above object, the present invention provides a method for automatically grouping logs based on text beginning and ending information, comprising:

[0010] Get the log text data to be processed;

[0011] Determine the beginning mark in the log text data to be processed; the beginning mark is used to identify the character string or symbol at the beginning of each group of logs;

[0012] Determine an end mark in the log text data to be processed; the end mark is used to identify a character string or symbol at the end of each group of logs;

[0013] Scan the content of the log text data to be processed, and use a string matching algorithm to determine the positions of all beginning tags and all ending tags in the log text data to be processed;

[0014] The log text data to be processed is divided into a plurality of log groups according to the positions of all beginning marks and all ending marks in the log text data to be processed.

[0015] Optionally, obtain the log text data to be processed, including:

[0016] Read the log text file to be processed;

[0017] The irrelevant contents in the log text file to be processed are removed to obtain the log text data to be processed; the irrelevant contents include metadata information, debugging information, repeated contents and format control characters.

[0018] Optionally, a regular expression matching method is used to remove metadata information, debugging information and repeated content in the log text file to be processed.

[0019] Optionally, determining the beginning mark in the log text data to be processed specifically includes:

[0020] Using a string matching algorithm to determine the repeated string in each line of the log text data to be processed;

[0021] Draw a position distribution map according to the position of the repeated character string in each line of the log text data to be processed;

[0022] The repeated character string that appears most frequently at the beginning of the log in the position distribution graph is used as the start mark.

[0023] Optionally, determining the end mark in the log text data to be processed specifically includes:

[0024] Using a string matching algorithm to determine the repeated string in each line of the log text data to be processed;

[0025] Draw a position distribution map according to the position of the repeated character string in each line of the log text data to be processed;

[0026] The repeated character string that appears most times at the end of the log in the position distribution graph is used as the end mark.

[0027] Optionally, scanning the content of the log text data to be processed and using a string matching algorithm to determine the positions of all beginning tags and all ending tags in the log text data to be processed specifically includes:

[0028] Sequentially scan each line of the log text data to be processed;

[0029] For any line of content in the log text data to be processed, the KMP algorithm is used to match the beginning mark and the ending mark in the line of content. If the beginning mark or the ending mark exists in the line of content, the position of the beginning mark or the ending mark in the line of content is recorded.

[0030] Optionally, the log text data to be processed is divided into a plurality of log groups according to the positions of all beginning tags and all ending tags in the log text data to be processed, specifically including:

[0031] All the lines between a set of matching start tags and end tags in the log text data to be processed are combined as a log group.

[0032] To achieve the above objectives, the present invention also provides a computer system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned method for automatic log grouping based on text beginning and ending information.

[0033] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention uses the beginning mark and the ending mark for matching, and can automatically complete the grouping of log text data without human intervention, which greatly improves the efficiency of log grouping and has high accuracy. In addition, the grouping of log text data is more flexible and can be widely applied to various types of log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A flowchart of a method for automatically grouping logs based on text beginning and ending information provided by the present invention.

[0036] Figure 2 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The purpose of the present invention is to provide a method and system for automatically grouping logs based on text start and end information, which utilizes universal start and end tags to achieve log grouping, can adapt to logs with varying formats, and can support cross-line recognition without manual intervention. It can automatically and efficiently parse the log structure, greatly improving the automation level and quality of log processing.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1 As shown, the method for automatically grouping logs based on text beginning and ending information provided by the present invention includes:

[0041] Step 100: Obtain log text data to be processed.

[0042] Specifically, the log text file to be processed is first read, and then irrelevant contents in the log text file to be processed are removed to obtain the log text data to be processed.

[0043] The irrelevant content includes:

[0044] (1) Metadata information: information that is not related to the actual content of the log, such as the log's fixed title, creation time, and the server where the log is located.

[0045] (2) Debug information: Information that is useless for log analysis, such as the program's debug output statements and other information that is only used for debugging.

[0046] (3) Duplicate content: Large sections of duplicate content often appear in logs, which are useless for analysis. It should be noted that the duplicate content removed here does not contain the grouping field (beginning mark and end mark). If the duplicate content contains the grouping field, it is judged as related content and will not be removed.

[0047] (4) Format control characters: such as line feed, carriage return and other format control characters.

[0048] In this embodiment, a regular expression matching method is used to remove metadata information, debugging information and repeated content in the log text file to be processed. All format control characters are deleted.

[0049] Metadata information usually appears at the beginning or end of a file and begins with a specific tag or comment. The regular expression for matching metadata information is: ^#.*|.*$.

[0050] Debug information is usually comments or output in the code for debugging purposes. The regular expression for matching debug information is: / / \s*debug.*.

[0051] Repeated content can be the same string or text that appears continuously. The regular expression for matching repeated content is: (.*)\1+.

[0052] Step 200: Determine the beginning mark in the log text data to be processed.

[0053] Specifically, the log text data to be processed is analyzed, and according to the structural characteristics of the logs, a character string or symbol used to identify the beginning of each group of logs is determined as a beginning mark, such as a timestamp, a specific keyword, etc.

[0054] The process of determining the opening tag includes:

[0055] (1) Read the log text data to be processed.

[0056] (2) Using a string matching algorithm to determine the repeated string in each line of the log text data to be processed. Specifically, statistically analyzing the line text of the log text data to be processed to identify frequently occurring repeated strings.

[0057] (3) Draw a position distribution map according to the position of the repeated character string in each line of the log text data to be processed, that is, analyze the position distribution law of the repeated character string and draw the position distribution map.

[0058] (4) The repeated string that appears most frequently at the beginning of the log in the position distribution graph is used as the start marker. Specifically, the regularly repeated string (such as "BEGIN" or "2022-03-0109:00:00") that appears at the beginning of the log is determined as the start marker.

[0059] Step 300: Determine the end mark in the log text data to be processed.

[0060] Specifically, the log text data to be processed is analyzed, and according to the structural characteristics of the logs, a character string or symbol used to identify the end of each group of logs is determined as an end mark, such as a timestamp, a specific keyword, etc.

[0061] The process of determining the end marker includes:

[0062] (1) Read the log text data to be processed.

[0063] (2) A string matching algorithm is used to determine the repeated string in each line of the log text data to be processed.

[0064] (3) Draw a position distribution diagram according to the position of the repeated character string in each line of the log text data to be processed.

[0065] (4) The repeated character string that appears most frequently at the end of the log in the position distribution graph is used as the end marker. Specifically, the character string (such as "END") that appears frequently at the end of the log in the position distribution graph is analyzed and used as the end marker.

[0066] Step 400: Scan the content of the log text data to be processed, and use a string matching algorithm to determine the positions of all beginning tags and all ending tags in the log text data to be processed.

[0067] Further, step 400 includes:

[0068] (1) Read the start and end tags of the definition.

[0069] (2) Sequentially scan each line of the log text data to be processed. As a specific implementation, the log text data to be processed may be scanned across multiple lines.

[0070] (3) For any line of content in the log text data to be processed, the KMP algorithm is used to match the start mark and the end mark in the line of content. If the line of content contains a start mark or an end mark, the position of the start mark or the end mark in the line of content is recorded.

[0071] The KMP algorithm is used to match the string representing the start tag from the beginning of the line. The specific matching process is as follows: a partial matching table for the start tag is constructed; the KMP algorithm is used to match the start tag based on the partial matching table. Similarly, the KMP algorithm is used to match the string representing the end tag from the end of the line. According to the matching result, whether the line contains the start tag or the end tag is recorded. This process continues until all the log text data to be processed completes the tag matching.

[0072] The above matching process can accurately identify the beginning and ending tags defined in the log, providing a basis for subsequent log grouping. The entire matching process fully utilizes the efficient string matching advantages of the KMP algorithm to achieve fast and accurate tag scanning.

[0073] Step 500: Divide the log text data to be processed into a plurality of log groups according to the positions of all beginning tags and all ending tags in the log text data to be processed.

[0074] Specifically, all lines between a set of matching start tags and end tags in the log text data to be processed are combined as a log group.

[0075] After obtaining multiple log groups, the text data containing the complete log groups is output in a traversal form to complete the automatic grouping processing of the entire log.

[0076] Compared with the existing log grouping method, the present invention has the following advantages:

[0077] (1) High degree of automation: The same set of log data is regarded as a paragraph. By defining specific start and end tags to identify the boundaries of the paragraph, and using the text start and end information for tag matching, log grouping can be completed automatically without human intervention, greatly improving work efficiency.

[0078] (2) Wide scope of application: The present invention only relies on common beginning and ending information for grouping, making log grouping more flexible. Therefore, it can be widely applied to various types of log data, especially to scenarios with complex content and non-fixed grouping marks, which significantly expands the scope of application.

[0079] (3) High accuracy. Since the grouping is based on specific start and end tags, the grouping result is very accurate and the error rate is very low.

[0080] (4) High processing performance: Scanning and identifying tags uses a simple and efficient method called string matching, so the log data can be processed very quickly.

[0081] (5) Support for cross-line recognition. The scanner can identify the beginning and ending tags across multiple lines, so it can also correctly handle cross-line log data, significantly improving the robustness of the method.

[0082] In summary, compared with the existing technology, the present invention is more flexible and intelligent while ensuring high efficiency, has a high degree of automation, a wide range of applicability, high accuracy, and a fast processing speed. It can effectively solve the problem of automatic classification and grouping of the growing amount of massive log data, greatly improve the efficiency and quality of log analysis, and reduce processing costs. It is an important progress in the field of log processing.

[0083] In one embodiment, a computer device is provided. The computer device may be a database, and its internal structure diagram may be as follows: Figure 2As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0084] In one embodiment, a computer system is also provided, including a memory and a processor, wherein a computer program is stored in the memory and can be run on the processor, and when the processor executes the computer program, the steps in the above-mentioned method for automatic log grouping based on text beginning and ending information are implemented.

[0085] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method for automatically grouping logs based on text beginning and ending information are implemented.

[0086] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for automatic log grouping based on text beginning and ending information.

[0087] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatically grouping logs based on the beginning and end information of the text. It is characterized in that The method for automatically grouping logs based on text beginning and ending information includes: Get the log text data to be processed; Determining the beginning mark in the log text data to be processed specifically includes: using a string matching algorithm to determine the repeated string in each line of the log text data to be processed; drawing a position distribution map according to the position of the repeated string in each line of the log text data to be processed; using the repeated string that appears the most times at the beginning position of the log in the position distribution map as the beginning mark; the beginning mark is used to identify the string or symbol at the beginning of each group of logs; Determine an end mark in the log text data to be processed; the end mark is used to identify a character string or symbol at the end of each group of logs; Scan the content of the log text data to be processed, and use a string matching algorithm to determine the positions of all beginning tags and all ending tags in the log text data to be processed, specifically including: Sequentially scan each line of the log text data to be processed; For any line of content of the log text data to be processed, the KMP algorithm is used to match the start mark and the end mark in the line content. If the start mark or the end mark exists in the line content, the position of the start mark or the end mark in the line content is recorded; wherein, the KMP algorithm is used to match the character string representing the start mark from the beginning of the line, and the specific matching process is: construct a partial matching table of the start mark; implement the KMP algorithm to match the start mark according to the partial matching table; similarly, the KMP algorithm is used to match the character string representing the end mark from the end of the line, and whether the line contains the start mark or the end mark is recorded according to the matching result; until all the log text data to be processed complete the tag matching; The log text data to be processed is divided into a plurality of log groups according to the positions of all beginning marks and all ending marks in the log text data to be processed.

2. The method for automatically grouping logs based on text beginning and ending information according to claim 1, It is characterized in that Get the log text data to be processed, including: Read the log text file to be processed; The irrelevant contents in the log text file to be processed are removed to obtain the log text data to be processed; the irrelevant contents include metadata information, debugging information, repeated contents and format control characters.

3. The method for automatically grouping logs based on text beginning and ending information according to claim 2, It is characterized in that The metadata information, debugging information and repeated contents in the log text file to be processed are removed by regular expression matching.

4. The method for automatically grouping logs based on text beginning and ending information according to claim 1, It is characterized in that Determining the end mark in the log text data to be processed specifically includes: Using a string matching algorithm to determine the repeated string in each line of the log text data to be processed; Draw a position distribution map according to the position of the repeated character string in each line of the log text data to be processed; The repeated character string that appears most times at the end of the log in the position distribution graph is used as the end mark.

5. The method for automatically grouping logs based on text beginning and ending information according to claim 1, It is characterized in that According to the positions of all beginning tags and all ending tags in the log text data to be processed, the log text data to be processed is divided into a plurality of log groups, specifically including: All the lines between a set of matching start tags and end tags in the log text data to be processed are combined as a log group.

6. A computer system, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method for automatic log grouping based on text beginning and ending information as described in any one of claims 1-5.

Images