Log field extraction method and device based on machine learning, equipment and medium

Abstract

The invention discloses a log field extraction method and device based on machine learning, equipment and a medium. The method comprises the steps of obtaining a basic historical log data set corresponding to a target log generation source; standardizing the basic historical log data set to obtain a target historical log data set, and analyzing and processing the target historical log data set based on a preset feature classification model to obtain a common mode set corresponding to a target log generation source; analyzing and processing the common mode set based on a preset regular expression, and generating a regular expression set corresponding to the target log generation source; and performing field matching on the current log data corresponding to the target log generation source based on the regular expression set to obtain a target field corresponding to the current log data. According to the technical scheme, log field extraction can be carried out on different log generation sources, and the accuracy and extraction efficiency of the log fields are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log field extraction method, device, equipment and medium based on machine learning. Background Art

[0002] In the computer field, logs generated by different log sources are generally agreed upon in different formats. However, since logs in different formats contain different information content, it is very difficult to analyze logs in different formats.

[0003] In the prior art, most log analysis systems rely on fixed parsing rules or require a lot of manual configuration to extract log fields from each log to adapt to different log formats. However, with the increase in data volume and the diversification of formats, if fixed parsing rules are used to extract log fields, the parsing of log data will lack flexibility and will be difficult to adapt to complex or irregular data formats; if manual configuration is used to extract log fields, the efficiency of field extraction will be reduced and errors will be prone to occur. Therefore, how to adapt to different log generation sources, extract log fields, and improve the accuracy and extraction efficiency of log fields is a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a log field extraction method, device, equipment and medium based on machine learning, which can solve the problems of low accuracy and extraction efficiency of log fields.

[0005] According to one aspect of the present invention, a log field extraction method based on machine learning is provided, comprising:

[0006] Obtain the basic historical log data set corresponding to the target log generation source;

[0007] The basic historical log data set is processed by standardization to obtain a target historical log data set, and the target historical log data set is analyzed and processed based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source;

[0008] Parsing and processing the common pattern set based on a preset regular expression to generate a regular expression set corresponding to the target log generation source;

[0009] Based on the regular expression set, field matching is performed on the current log data corresponding to the target log generation source to obtain the target field corresponding to the current log data.

[0010] According to another aspect of the present invention, a log field extraction device based on machine learning is provided, comprising:

[0011] The data acquisition module is used to obtain the basic historical log data set corresponding to the target log generation source;

[0012] A pattern determination module, used for standardizing the basic historical log data set to obtain a target historical log data set, and analyzing and processing the target historical log data set based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source;

[0013] A regular expression construction module, used to parse and process the common pattern set based on a preset regular expression to generate a regular expression set corresponding to a target log generation source;

[0014] The field extraction module is used to perform field matching on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log field extraction method based on machine learning described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log field extraction method based on machine learning described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the log field extraction method based on machine learning described in any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention obtains the target historical log data set by standardizing the basic historical log data set corresponding to the target log generation source, and analyzes and processes the target historical log data set based on a preset feature classification model to obtain a set of common patterns corresponding to the target log generation source. Furthermore, the common pattern set is parsed and processed based on a preset regular expression to generate a set of regular expressions corresponding to the target log generation source. Finally, field matching is performed on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data. By analyzing the log features of different log generation sources, the problem that the existing field extraction method cannot adapt to different log generation sources is solved, and log fields can be extracted from logs generated by different log generation sources, thereby improving the accuracy and extraction efficiency of log fields.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a log field extraction method based on machine learning provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a log field extraction method based on machine learning provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a structural schematic diagram of a log field extraction device based on machine learning provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing the log field extraction method based on machine learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "target", "original", "base", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 The present invention provides a flowchart of a method for extracting log fields based on machine learning in the first embodiment of the present invention. This embodiment is applicable to the case of extracting log fields from different log generation sources. The method can be performed by a log field extraction device based on machine learning. The log field extraction device based on machine learning can be implemented in the form of hardware and / or software. The log field extraction device based on machine learning can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110: Obtain a basic historical log data set corresponding to a target log generation source.

[0033] The log generation source may refer to a source end for generating log data. Exemplarily, the log generation source may be an application server, a database, or a system operation log, etc. The target log generation source may refer to a log generation source selected for field extraction. It is worth noting that in the embodiment of the present invention, the number of log generation sources included in the target log generation source is at least one.

[0034] The historical log data may refer to the log data generated by the target log generation source in the historical time period. The basic historical log data may refer to the historical log data after data preprocessing. The basic historical log data set may refer to the set of basic historical log data corresponding to the same target log generation source.

[0035] In an optional embodiment, before obtaining the basic historical log data set corresponding to the target log generation source, it may also include: obtaining the initial historical log data set corresponding to the target log generation source; performing data preprocessing on the initial historical log data set to obtain the basic historical log data set corresponding to the target log generation source. Among them, the initial historical log data may refer to the original historical log data that has not been preprocessed. The initial historical log data set may refer to a set of initial historical log data corresponding to the same target log generation source. Data preprocessing may refer to the operation of cleaning the historical log data. Exemplarily, data preprocessing may include removing useless information, correcting erroneous formats, and unifying timestamp representations.

[0036] Specifically, after selecting the need to perform field extraction on the log data generated by the target log generation source, the initial historical log data set generated by the target log generation source in the historical time period can be obtained, for example, "2024-08-0810:12:34,123ERROR[user-service]-User ID 12345:Invalid login attempt" and "2024-08-0810:13:45,678INFO[payment-service]-Payment transaction ID 98765:Completedsuccessfully". Afterwards, data preprocessing is performed on the collected initial historical log data set. As a result, a basic historical log data set is obtained that only retains useful information, has a correct format, and has a unified timestamp representation, such as "[2024-08-0810:12:34,123][ERROR][user-service][User ID 12345:Invalid login attempt]" and "[2024-08-0810:13:45,678][INFO][payment-service][Payment transaction ID 98765:Completed successfully]", providing a valid basis for subsequent operations.

[0037] It is worth noting that after the basic historical log data set is generated, the basic historical log data set can be stored in a memory, so that subsequent operations can directly obtain the basic historical log data set from the memory, thereby improving the efficiency of subsequent work.

[0038] S120 , standardize the basic historical log data set to obtain a target historical log data set, and analyze and process the target historical log data set based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source.

[0039] Among them, standardization processing may refer to the operation of performing secondary data processing on the basic historical log data in the basic historical log data set. Exemplarily, standardization processing may include word segmentation or removal of useless characters, etc. The target historical log data may refer to the log data obtained after standardization processing of the basic historical log data. The target historical log data set may refer to a set of target historical log data corresponding to the same target log generation source.

[0040] Among them, the preset feature classification model may refer to a preset model for feature classification of log data. In an optional embodiment, the preset feature classification model may include: a preset feature extraction model and a preset clustering model. Among them, the preset feature extraction model may refer to a preset model for feature extraction of log data. Exemplarily, the preset feature extraction model may be a term frequency-inverse document frequency (Term Frequency–Inverse Document Frequency, TF-IDF) algorithm model. The preset clustering model may refer to a preset model for clustering feature extraction results. Exemplarily, the preset clustering model may be a K-means algorithm model.

[0041] Among them, the common pattern may refer to a pattern with the same field format. Generally, a common pattern includes various target historical log data with the same field format. The common pattern set may refer to a set composed of various common patterns corresponding to the same target log generation source.

[0042] S130: parsing and processing the common pattern set based on a preset regular expression to generate a regular expression set corresponding to a target log generation source.

[0043] Among them, the preset regular expression may refer to a pre-set regular expression. Exemplarily, the preset regular expression may be an existing known regular expression. Generally, the field type of each log data can be determined according to the preset regular expression. Parsing processing may refer to the operation of determining the field type of log data. The regular expression set may refer to a set composed of regular expressions constructed according to a common pattern set.

[0044] In an optional embodiment, after the common pattern set is parsed and processed based on a preset regular expression to generate a regular expression set corresponding to the target log generation source, it may also include: obtaining the log data to be tested and the corresponding expected field results corresponding to the target log generation source; performing field matching on the log data to be tested based on the regular expression set to obtain the basic field results corresponding to the log data to be tested; performing a difference comparison between the basic field results and the expected field results to generate a comparison result, and adjusting and processing the regular expression set based on the comparison result.

[0045] Among them, the log data to be tested may refer to the log data used for test processing. The expected field result may refer to the field parsing result that the user expects to obtain. Field matching may refer to the operation of extracting fields using a set of regular expressions. The basic field result may refer to the extracted field obtained after field matching of the log data to be tested. The comparison result may refer to the processing result obtained after comparing the basic field result with the expected field result. Exemplarily, the comparison result may be relatively consistent, or it may be relatively inconsistent and the corresponding error type, such as capture group misalignment, data type error, or uncaptured field. Adjustment processing may refer to the operation of adjusting the regular expression corresponding to the inconsistent comparison result. Specifically, the specific content of the adjustment processing can be determined according to the corresponding error type. Exemplarily, the adjustment processing may be to expand the matching range of a capture group, or to modify the data type definition of a field.

[0046] Specifically, after generating a regular expression set corresponding to the target log generation source, the regular expression set can be optimized and adjusted using the log data to be tested and the corresponding expected field results corresponding to the target log generation source, thereby improving the accuracy and efficiency of log parsing.

[0047] It is worth noting that after optimizing and adjusting the regular expression set, the updated regular expression rules need to be verified through a wider data set to ensure that the modification does not introduce new errors. Once verified, the optimized and adjusted regular expression set is used as a new regular expression set for subsequent operations. In addition, during the use of the regular expression set, the performance of the regular expression set can also be monitored using a predefined test set containing annotated samples to ensure that the regular expression set remains efficient and accurate when processing actual log data.

[0048] S140. Perform field matching on current log data corresponding to the target log generation source based on the regular expression set to obtain a target field corresponding to the current log data.

[0049] The current log data may refer to the log data that needs to be subjected to field extraction at the current moment. The target field may refer to the field extraction result corresponding to the current log data. Exemplarily, the target field may include each data field included in the current log data, and the field type corresponding to each data field.

[0050] The technical solution of the embodiment of the present invention obtains the target historical log data set by standardizing the basic historical log data set corresponding to the target log generation source, and analyzes and processes the target historical log data set based on a preset feature classification model to obtain a set of common patterns corresponding to the target log generation source. Furthermore, the common pattern set is parsed and processed based on a preset regular expression to generate a set of regular expressions corresponding to the target log generation source. Finally, field matching is performed on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data. By analyzing the log features of different log generation sources, the problem that the existing field extraction method cannot adapt to different log generation sources is solved, and log fields can be extracted from logs generated by different log generation sources, thereby improving the accuracy and extraction efficiency of log fields.

[0051] Embodiment 2

[0052] Figure 2 A flowchart of a log field extraction method based on machine learning is provided in the second embodiment of the present invention. This embodiment is refined based on the above-mentioned embodiment. In this embodiment, the operation of analyzing and processing the target historical log data set based on a preset feature classification model to obtain a set of common patterns corresponding to the target log generation source is specifically refined. Specifically, it may include: extracting features from the target historical log data set based on a preset feature extraction model to obtain target feature values ​​corresponding to each target historical log data in the target historical log data set; clustering the target feature values ​​corresponding to each target historical log data based on a preset clustering model to obtain a set of clustering results corresponding to the target historical log data set as the set of common patterns corresponding to the target log generation source. Figure 2 As shown, the method includes:

[0053] S210: Obtain a basic historical log data set corresponding to a target log generation source.

[0054] S220 , standardize the basic historical log data set to obtain a target historical log data set.

[0055] Specifically, after collecting the basic historical log data set corresponding to the target log generation source, the basic historical log data in the basic historical log data set can be standardized to obtain the standardized target historical log data, and the target historical log data corresponding to the same basic historical log data set can be combined to obtain the target historical log data set, providing an effective basis for subsequent operations.

[0056] S230: Perform feature extraction on the target historical log data set based on a preset feature extraction model to obtain target feature values ​​corresponding to each target historical log data in the target historical log data set.

[0057] The target feature value may refer to a feature extraction result corresponding to the target historical log data. For example, if the preset feature extraction model is a TF-IDF algorithm model, the target feature value may be a TF-IDF value corresponding to the target historical log data.

[0058] S240 , clustering the target feature values ​​corresponding to each target historical log data set based on a preset clustering model to obtain a clustering result set corresponding to the target historical log data set as a common pattern set corresponding to the target log generation source.

[0059] The clustering result may refer to the data result obtained after clustering all the target historical log data in the target historical log data set. Exemplarily, each clustering result includes each target historical log data of the same clustering category. The clustering result set may refer to the set composed of each clustering result corresponding to the same target historical log data set.

[0060] Specifically, after obtaining the target historical log data set, the preset feature extraction model can be used to extract features from each target historical log data in the target historical log data set to obtain the target feature value corresponding to each target historical log data. Afterwards, the preset clustering model is used to perform clustering processing on the target feature value corresponding to each target historical log data to obtain each clustering result corresponding to the target historical log data set as the common pattern corresponding to the target log generation source. In addition, the pattern features of the target historical log data in each common pattern are analyzed as the pattern category of the common pattern. For example, the target historical log data set is {"[2024-08-0810:12:34,123][ERROR][user-service][User ID 12345:Invalid login attempt]","[2024-08-0810:13:45,678][INFO][payment-service][Payment transaction ID 98765:Completedsuccessfully]","[2024-08-08 10:14:56,234][DEBUG][order-service][Order ID54321:Processing order]","[2024-08-0810:15:01,456][WARN][user-service][UserID 12346:Login attempt successful]","[2024-08-0810:16:02,789][INFO][payment-service][Payment transaction ID 12321:Payment failed]"} as an example, the preset feature extraction model is used to extract features from each target historical log data in the target historical log data set, and the target feature value corresponding to each target historical log data is obtained. After the target feature value corresponding to each target historical log data is clustered using the preset clustering model, the common patterns obtained can be: Common pattern 0: {1.[2024-08-0810:12:34,123][ERROR][user-service][User ID 12345:Invalid login attempt]; 2.[2024-08-0810:15:01,456][WARN][user-service][User ID 12346:Login attempt successful]}; Common pattern 1: {1.[2024-08-08 10:13:45,678][INFO][payment-service][Paymenttransaction ID 98765:Completed successfully]; 2.[2024-08-0810:16:02,789][INFO][payment-service][Payment transaction ID 12321:Payment failed]}; and, common pattern 2: {1.[2024-08-0810:14:56,234][DEBUG][order-service][Order ID 54321:Processingorder]}. Finally, combine and process the common patterns corresponding to the target log generation source as the common pattern set corresponding to the target log generation source. .

[0061] S250. Parsing and processing the target historical log data corresponding to the target common pattern in the common pattern set based on a preset regular expression to obtain a common feature vector corresponding to the target common pattern; wherein the common feature vector includes each basic common feature and key common feature corresponding to the same target common pattern.

[0062] Among them, the target common pattern may refer to the common pattern selected for parsing. The key common feature may refer to the field feature that represents the pattern category in the target common pattern. Exemplarily, the key common feature may be the field name that represents the pattern category. The basic common feature may refer to the remaining features in the target common pattern except the key common feature. Exemplarily, the basic common feature may be the remaining field name except the field name that represents the pattern category. The common feature vector may refer to a vector set composed of each basic common feature and key common feature corresponding to the same target common pattern.

[0063] Specifically, after generating a common pattern set, the existing preset regular expression can be used to parse each target historical log data in the target common pattern to obtain the field name corresponding to each target historical log data. Exemplarily, continuing with the above example, if the target common pattern is common pattern 0, the parsing result can be obtained: Timestamp: [2024-08-0810:12:34,123], [2024-08-0810:15:01,456]; Log Level: [ERROR], [WARN]; Service: [user-service]; User ID: User ID 12345, User ID 12346; Message: Invalid login attempt, Login attempt successful. Similarly, if the target common mode is common mode 1, the parsing result can be obtained: Timestamp: [2024-08-0810:13:45,678], [2024-08-0810:16:02,789]; Log Level: [INFO]; Service: [payment-service]; Transaction ID: Payment transaction ID 98765, Payment transaction ID 12321; Message: Completed successfully, Payment failed. If the target common mode is common mode 2, the parsing result can be obtained: Timestamp: [2024-08-0810:14:56,234]; Log Level: [DE BUG]; Service: [order-service]; Order ID: Order ID 54321; Message: Processing order.

[0064] Afterwards, each basic common feature and key common feature in each target common pattern is extracted to obtain a common feature vector corresponding to the target common pattern. Exemplarily, following the above example, if the target common pattern is common pattern 0, the basic common features may be: Timestamp, Service, Log Level and Message; the key common feature may be: User ID; and the common feature vector is: {Timestamp, Service, User ID, Message, Log Level}. Similarly, if the target common pattern is common pattern 1, the basic common features may be: Timestamp, Service, Log Level and Message; the key common feature may be: Transaction ID; and the common feature vector is: {Timestamp, Service, Transaction ID, Message, Log Level}. If the target common pattern is common pattern 2, the basic common features may be: Timestamp, Service, Log Level and Message; the key common feature may be: Order ID; and the common feature vector is: {Timestamp, Service, Order ID, Message, Log Level}.

[0065] S260 , processing each common feature vector based on the pattern category combination of the target common pattern to generate a regular expression set corresponding to the target log generation source.

[0066] The pattern category may refer to the category feature corresponding to the target common pattern. Generally, different target common patterns correspond to different pattern categories. Exemplarily, the pattern category of each target common pattern may be represented according to the key common features in the common feature vector.

[0067] Specifically, after obtaining the common feature vector corresponding to the target common pattern, the regular expression corresponding to each pattern category can be constructed according to the preset expression generation rules and the common feature vector. For example, if the common feature vector is {Timestamp, Service, User ID, Message, Log Level}, the corresponding regular expression can be: \[(?P <timestamp>.*?)\]\[(?P<log_level>.*?)\]\[(?P <service>.*?)\]\[User ID(?P<user_id>\d+):(?P <message>.*? )\]. Similarly, if the common feature vector is {Timestamp,Service,Transaction ID,Message,Log Level}, the corresponding regular expression can be: \[(?P <timestamp>.*?)\]\[(?P<log_level>.*?)\]\[(?P <service>.*?)\]\[Payment transaction ID(?P<transaction_id>\d+):(?P <message>.*? )\]. If the common feature vector is {Time stamp,Service,Order ID,Message,Log Level}, the corresponding regular expression can be: \[(?P <timestamp>.*?)\]\[(?P<log_level>.*?)\]\[(?P <service>.*?)\]\[Order ID(?P<order_id>\d+):(?P <message>.*? )\]. After that, the constructed regular expressions are combined and processed according to the pattern category of the target common pattern, thereby generating a regular expression set corresponding to the target log generation source, providing an effective basis for subsequent operations.

[0068] It is worth noting that in the embodiment of the present invention, after generating the regular expression set corresponding to the target log generation source, the regular expression set can be stored in the memory, so that subsequent operations can directly obtain the regular expression set from the memory, thereby improving the efficiency of subsequent work. In addition, to ensure storage efficiency, the memory storing the regular expression set may be inconsistent with the memory storing the basic historical log data set, which is not specifically limited in the embodiment of the present invention.

[0069] S270: Acquire current log data corresponding to the target log generation source, and perform field analysis on the current log data to determine key field features corresponding to the current log data.

[0070] Field parsing may refer to an operation of parsing log data. Field parsing may be used to determine the names of various fields in the log data. Key field features may refer to field names that represent data categories in the current log data.

[0071] S280: Determine a target regular expression corresponding to the current log data in the regular expression set based on the key field feature.

[0072] The target regular expression may refer to a regular expression in the regular expression set whose key common features are consistent with the key field features. Exemplarily, a match or search method may be used to perform matching to determine the target regular expression.

[0073] Specifically, after obtaining the current log data corresponding to the target log generation source, the current log data can be firstly parsed to obtain the field names corresponding to the current log data. Exemplarily, taking the current log data as "[2024-08-0810:15:01,456][WARN][user-service][User ID 12346:Login attemptsuccessful]" as an example, the field names corresponding to the current log data can be obtained as: timestamp, log level, service, User ID and Message. Afterwards, the key field features corresponding to the current log data are determined among all the field names. Exemplarily, following the above example, the key field features can be User ID. Further, a target regular expression whose key common features are consistent with the key field features is determined in the regular expression set. Exemplarily, following the above example, the target regular expression can be the regular expression corresponding to the common pattern 0: \[(?P <timestamp>.*?)\]\[(?P<log_level>.*?)\]\[(?P <service>.*?)\]\[User ID(?P<user_id>\d+):(?P <message>.*? )\]. In this way, the target regular expression corresponding to the current log data can be determined, which provides an effective basis for subsequent field extraction and improves the accuracy and extraction efficiency of log fields.

[0074] S290: Perform field matching on the current log data based on the target regular expression to obtain a target field corresponding to the current log data.

[0075] Specifically, after determining the target regular expression corresponding to the current log data, the target regular expression can be used to match the fields of the current log data to obtain the target field corresponding to the current log data. For example, following the above example, if the current log data is "[2024-08-0810:15:01,456][WARN][user-service][User ID12346:Login attempt successful]", the target regular expression is: \[(?P <timestamp>.*?)\]\[(?P<log_level>.*?)\]\[(?P <service>.*?)\]\[User ID(?P<user_id>\d+):(?P <message>.*? )\], the generated target field can be: {"timestamp":"2024-08-0810:15:01,456","loglevel":"WARN","service":"user-service","user id":"12346","message":"Loginattempt successful"}.

[0076] The technical solution of the embodiment of the present invention obtains the target historical log data set by standardizing the basic historical log data set corresponding to the target log generation source. After that, feature extraction is performed on the target historical log data set based on a preset feature extraction model to obtain the target feature value corresponding to each target historical log data in the target historical log data set, and the target feature value corresponding to each target historical log data is clustered based on a preset clustering model to obtain a clustering result set corresponding to the target historical log data set as the common pattern set corresponding to the target log generation source. Furthermore, the target historical log data corresponding to the target common pattern in the common pattern set is parsed based on a preset regular expression to obtain the common feature vector corresponding to the target common pattern, and each common feature vector is combined based on the pattern category of the target common pattern to generate a regular expression set corresponding to the target log generation source. Further, field parsing is performed on the current log data corresponding to the target log generation source to determine the key field features corresponding to the current log data, and the target regular expression corresponding to the current log data is determined in the regular expression set based on the key field features. Finally, field matching is performed on the current log data based on the target regular expression to obtain the target field corresponding to the current log data. By analyzing the log features of different log generation sources, the problem that the existing field extraction method cannot adapt to different log generation sources is solved. Log fields can be extracted from logs generated by different log generation sources, thereby improving the accuracy and extraction efficiency of log fields.

[0077] Embodiment 3

[0078] Figure 3 This is a schematic diagram of the structure of a log field extraction device based on machine learning provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data acquisition module 310, a pattern determination module 320, a regular construction module 330 and a field extraction module 340;

[0079] The data acquisition module 310 is used to acquire the basic historical log data set corresponding to the target log generation source;

[0080] The pattern determination module 320 is used to standardize the basic historical log data set to obtain a target historical log data set, and analyze and process the target historical log data set based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source;

[0081] A regular expression construction module 330 is used to parse and process the common pattern set based on a preset regular expression to generate a regular expression set corresponding to a target log generation source;

[0082] The field extraction module 340 is used to perform field matching on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data.

[0083] The technical solution of the embodiment of the present invention obtains the target historical log data set by standardizing the basic historical log data set corresponding to the target log generation source, and analyzes and processes the target historical log data set based on a preset feature classification model to obtain a set of common patterns corresponding to the target log generation source. Furthermore, the common pattern set is parsed and processed based on a preset regular expression to generate a set of regular expressions corresponding to the target log generation source. Finally, field matching is performed on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data. By analyzing the log features of different log generation sources, the problem that the existing field extraction method cannot adapt to different log generation sources is solved, and log fields can be extracted from logs generated by different log generation sources, thereby improving the accuracy and extraction efficiency of log fields.

[0084] Optionally, the preset feature classification model includes: a preset feature extraction model and a preset clustering model.

[0085] Optionally, the mode determination module 320 may be specifically used for:

[0086] Perform feature extraction on the target historical log data set based on a preset feature extraction model to obtain target feature values ​​corresponding to each target historical log data in the target historical log data set;

[0087] The target feature values ​​corresponding to each target historical log data are clustered based on a preset clustering model to obtain a clustering result set corresponding to the target historical log data set as a common pattern set corresponding to the target log generation source.

[0088] Optionally, the regular construction module 330 may be specifically used for:

[0089] Parsing and processing the target historical log data corresponding to the target common pattern in the common pattern set based on a preset regular expression to obtain a common feature vector corresponding to the target common pattern; wherein the common feature vector includes each basic common feature and key common feature corresponding to the same target common pattern;

[0090] The common feature vectors are processed based on the pattern category combination of the target common pattern to generate a regular expression set corresponding to the target log generation source.

[0091] Optionally, the field extraction module 340 may be specifically used for:

[0092] Obtaining current log data corresponding to the target log generation source, and performing field analysis on the current log data to determine key field features corresponding to the current log data;

[0093] Determine a target regular expression corresponding to the current log data in the regular expression set based on the key field feature;

[0094] Field matching is performed on the current log data based on the target regular expression to obtain a target field corresponding to the current log data.

[0095] Optionally, the log field extraction device based on machine learning may also include: an adjustment processing module, which is used to obtain the log data to be tested and the corresponding expected field results corresponding to the target log generation source after the common pattern set is parsed and processed based on the preset regular expression to generate the regular expression set corresponding to the target log generation source; perform field matching on the log data to be tested based on the regular expression set to obtain the basic field results corresponding to the log data to be tested; perform difference comparison between the basic field results and the expected field results to generate a comparison result, and adjust and process the regular expression set based on the comparison result.

[0096] Optionally, the log field extraction device based on machine learning may also include: a data preprocessing module, used to obtain an initial historical log data set corresponding to the target log generation source before obtaining the basic historical log data set corresponding to the target log generation source; and perform data preprocessing on the initial historical log data set to obtain a basic historical log data set corresponding to the target log generation source.

[0097] The machine learning-based log field extraction device provided in the embodiment of the present invention can execute the machine learning-based log field extraction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0098] Embodiment 4

[0099] Figure 4 A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0100] like Figure 4 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0101] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The processor 420 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 420 performs the various methods and processes described above, such as a log field extraction method based on machine learning.

[0103] The method includes:

[0104] Obtain the basic historical log data set corresponding to the target log generation source;

[0105] The basic historical log data set is processed by standardization to obtain a target historical log data set, and the target historical log data set is analyzed and processed based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source;

[0106] Parsing and processing the common pattern set based on a preset regular expression to generate a regular expression set corresponding to the target log generation source;

[0107] Based on the regular expression set, field matching is performed on the current log data corresponding to the target log generation source to obtain the target field corresponding to the current log data.

[0108] In some embodiments, the log field extraction method based on machine learning may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the log field extraction method based on machine learning described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the log field extraction method based on machine learning in any other appropriate manner (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] The present application also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the log field extraction method based on machine learning provided in any embodiment of the present application. The program product and the log field extraction method based on machine learning disclosed in each embodiment of the present application belong to the same inventive concept, so it will not be described here.

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / message> < / service> < / timestamp> < / message> < / service> < / timestamp> < / message> < / service> < / timestamp> < / message> < / service> < / timestamp> < / message> < / service> < / timestamp>

Claims

1. A log field extraction method based on machine learning, characterized in that: include: Obtain the basic historical log data set corresponding to the target log generation source; The basic historical log data set is processed by standardization to obtain a target historical log data set, and the target historical log data set is analyzed and processed based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source; Parsing and processing the common pattern set based on a preset regular expression to generate a regular expression set corresponding to the target log generation source; Based on the regular expression set, field matching is performed on the current log data corresponding to the target log generation source to obtain the target field corresponding to the current log data.

2. The method according to claim 1, characterized in that The preset feature classification model includes: a preset feature extraction model and a preset clustering model.

3. The method according to claim 2, characterized in that The analyzing and processing the target historical log data set based on the preset feature classification model to obtain a set of common patterns corresponding to the target log generation source includes: Perform feature extraction on the target historical log data set based on a preset feature extraction model to obtain target feature values ​​corresponding to each target historical log data in the target historical log data set; The target feature values ​​corresponding to each target historical log data are clustered based on a preset clustering model to obtain a clustering result set corresponding to the target historical log data set as a common pattern set corresponding to the target log generation source.

4. The method according to claim 1, characterized in that The process of parsing and processing the common pattern set based on a preset regular expression to generate a regular expression set corresponding to a target log generation source includes: Parsing and processing the target historical log data corresponding to the target common pattern in the common pattern set based on a preset regular expression to obtain a common feature vector corresponding to the target common pattern; wherein the common feature vector includes each basic common feature and key common feature corresponding to the same target common pattern; The common feature vectors are processed based on the pattern category combination of the target common pattern to generate a regular expression set corresponding to the target log generation source.

5. The method according to claim 1, characterized in that The performing field matching on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data includes: Obtaining current log data corresponding to the target log generation source, and performing field analysis on the current log data to determine key field features corresponding to the current log data; Determine a target regular expression corresponding to the current log data in the regular expression set based on the key field feature; Field matching is performed on the current log data based on the target regular expression to obtain a target field corresponding to the current log data.

6. The method according to claim 1, characterized in that After the common pattern set is parsed and processed based on the preset regular expression to generate a regular expression set corresponding to the target log generation source, the method further includes: Obtain the log data to be tested and the corresponding expected field results corresponding to the target log generation source; Performing field matching on the log data to be tested based on the regular expression set to obtain basic field results corresponding to the log data to be tested; The basic field result and the expected field result are difference compared to generate a comparison result, and the regular expression set is adjusted and processed based on the comparison result.

7. The method according to claim 1, characterized in that Before obtaining the basic historical log data set corresponding to the target log generation source, the method further includes: Obtain the initial historical log data set corresponding to the target log generation source; The initial historical log data set is subjected to data preprocessing to obtain a basic historical log data set corresponding to a target log generation source.

8. A log field extraction device based on machine learning, characterized in that: include: The data acquisition module is used to obtain the basic historical log data set corresponding to the target log generation source; A pattern determination module, used for standardizing the basic historical log data set to obtain a target historical log data set, and analyzing and processing the target historical log data set based on a preset feature classification model to obtain a common pattern set corresponding to the target log generation source; A regular expression construction module, used to parse and process the common pattern set based on a preset regular expression to generate a regular expression set corresponding to a target log generation source; The field extraction module is used to perform field matching on the current log data corresponding to the target log generation source based on the regular expression set to obtain the target field corresponding to the current log data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log field extraction method based on machine learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log field extraction method based on machine learning as described in any one of claims 1 to 7 when executed.

Images