Training sample generation, model training, extraction methods, equipment, media and products

Abstract

This invention discloses a method, device, medium, and product for generating training samples, training models, and extracting data, relating to the field of computer technology. The method includes: acquiring raw log data and standardizing each raw log data set to obtain a first log dataset; performing pattern mining on each log data set in the first log dataset to obtain a log data pattern set; determining the field structure types contained in each log data pattern, extracting the variable field content from each field structure type and naming the variables to obtain a mapping table between fields and variables; generating training samples based on the raw log data, the log data pattern set, and the mapping table between fields and variables; and using each training sample to train a field extraction model for log data. The solution of this invention can quickly and accurately generate training samples, providing a basis for training a field extraction model for log data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, device, medium, and product for generating training samples, training models, and extracting training samples. Background Technology

[0002] With the rapid development of information technology, people generate a large amount of unstructured data in various formats in various social and economic activities. To convert unstructured data into structured data, the information in the logs needs to be classified and extracted first.

[0003] Currently, the acquired log data is usually manually labeled, and then the corresponding model is trained based on the labeled log data to extract relevant log fields. This method relies entirely on the accuracy of the manually labeled data, which is inaccurate, slow, and consumes a lot of human resources.

[0004] How to quickly and accurately generate training samples to provide a basis for training field extraction models from log data and reduce the consumption of human resources is a key research issue in the industry. Summary of the Invention

[0005] This invention provides a training sample generation, model training, extraction method, device, medium, and product to quickly and accurately generate training samples, providing a basis for training field extraction models of log data and reducing the consumption of human resources.

[0006] According to one aspect of the present invention, a method for generating training samples is provided, the method comprising:

[0007] Obtain raw log data and standardize each piece of raw log data to obtain the first log dataset;

[0008] Pattern mining is performed on each log data in the first log dataset to obtain a log data pattern set; where each log data pattern corresponds to a log data category.

[0009] Determine the field structure types included in each log data pattern, extract the variable field content in each field structure type and name the variables to obtain a mapping relationship table between fields and variables;

[0010] Training samples are generated based on the original log data, the log data pattern set, and the mapping relationship table between fields and variables; each training sample is used to train the field extraction model of the log data.

[0011] According to another aspect of the present invention, a method for training a field extraction model for log data is provided, comprising:

[0012] Determine the training samples; the training samples are generated by the training sample generation method described in any embodiment of the present invention;

[0013] The training samples are input into the target machine learning model for iterative training. When the iteration stops, the field extraction model of the log data is obtained.

[0014] According to another aspect of the present invention, a method for extracting fields from log data is provided, comprising:

[0015] Obtain target log data;

[0016] The target log data is input into the field extraction model of the log data to obtain the target field that matches the target log and the field value that matches the target field;

[0017] The field extraction model for log data is trained using the training method for the field extraction model of log data described in any embodiment of this invention.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the training sample generation method according to any embodiment of the present invention, or the training of the field extraction model of log data, or the field extraction method of log data.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the training sample generation method, or the training of a field extraction model for log data, or the field extraction method for log data as described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the training sample generation method, the training of a field extraction model for log data, or the field extraction method for log data as described in any embodiment of the present invention.

[0024] The technical solution of this invention involves acquiring raw log data and standardizing it to obtain a first log dataset; performing pattern mining on each log data in the first log dataset to obtain a log data pattern set; wherein each log data pattern corresponds to a log data category; determining the field structure type contained in each log data pattern, extracting the variable field content in each field structure type and naming the variables to obtain a mapping relationship table between fields and variables; generating training samples based on the raw log data, the log data pattern set, and the mapping relationship table between fields and variables; and using the training samples to train the field extraction model of log data, which can quickly and accurately generate training samples, providing a basis for training the field extraction model of log data and reducing the consumption of human resources.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a training sample generation method provided in Embodiment 1 of the present invention;

[0028] Figure 2 This is a flowchart of a method for generating training samples according to Embodiment 2 of the present invention;

[0029] Figure 3 This is a flowchart of a training method for a field extraction model of log data according to Embodiment 3 of the present invention;

[0030] Figure 4 This is a flowchart of a method for extracting fields from log data according to Embodiment 4 of the present invention;

[0031] Figure 5 This is a schematic diagram of a training sample generation device according to Embodiment 5 of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of a training device for a field extraction model of log data according to Embodiment Six of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of a field extraction device for log data provided in Embodiment 7 of the present invention;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the method for generating training samples, training a field extraction model for log data, or a method for extracting field data from log data, as described in the embodiments of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] Figure 1 This is a flowchart of a training sample generation method according to Embodiment 1 of the present invention. This embodiment is applicable to the case of automatically generating training samples for a field extraction model of log data. This method can be executed by a training sample generation device, which can be implemented in hardware and / or software, and can be configured in electronic devices such as computers, servers, or tablet computers. Figure 1 As shown, the method includes:

[0039] Step 110: Obtain raw log data and standardize each piece of raw log data to obtain the first log dataset.

[0040] Log data refers to procedural event records generated during the operation of servers, network devices, security devices, databases, middleware, and business systems in an Internet production environment. Log data is unstructured text, and the log information in the log data records specific system behaviors through timestamps, log levels, and log content.

[0041] Optionally, in this embodiment, the original log dataset can be obtained from the target storage. It should be noted that the original log data involved in this embodiment can be log data generated by different data sources, and the categories of each log data can be the same or different. Among them, the data source can be various types of databases, files or services, etc., and this embodiment does not limit them.

[0042] Optionally, in this embodiment, after obtaining the original log data, the obtained log data can be further standardized to obtain the first log dataset.

[0043] In one optional implementation of this embodiment, after obtaining the original log data, the obtained original log data can be further cleaned to filter out incomplete or duplicate log data in order to remove irrelevant information and noise; furthermore, the cleaned log data can be formatted uniformly for subsequent processing.

[0044] It should be noted that the log data involved in this embodiment was obtained only after user authorization, and the method of acquisition is reasonable and legal.

[0045] Step 120: Perform pattern mining on each log data in the first log dataset to obtain a log data pattern set.

[0046] Each log data pattern corresponds to a log data category.

[0047] Optionally, in this embodiment, after obtaining the first log dataset, pattern mining can be further performed on each log data in the first log dataset to obtain each log data pattern corresponding to different log data categories. In this embodiment, these log data patterns are referred to as the log data pattern set. It should be noted that the number of log data patterns can be 3, 5, or 10, etc., and this embodiment does not limit it.

[0048] Optionally, in this embodiment, performing pattern mining on each log data in the first log dataset to obtain a log data pattern set may include: splitting each log data into a word sequence based on a preset word segmentation character; setting up a multi-way tree and inserting each word sequence into each node of the multi-way tree; clustering each node based on the similarity of each word sequence to obtain each clustering result; extracting the stable part of the log data in each clustering result and replacing the dynamically changing content with the target identifier to obtain the log data pattern set.

[0049] The preset word segmentation character can be a space or a punctuation mark (e.g., a comma, an equal sign, or a colon), etc., and is not limited to it in this embodiment.

[0050] In this embodiment, each log data can be split into a sequence of tokens based on a preset token. For example, if the first log data is “Accepted password for alice from 192.168.1.1port22ssh2”, then the segmented log data can be “['Accepted','password','for','alice','from','192.168.1.1','port','22','ssh2']”.

[0051] The multi-branch tree can be a Drain search tree or other tree structures; this embodiment does not limit its specific type.

[0052] Furthermore, each of the aforementioned word sequences can be inserted into each node of the multi-way tree, and the nodes can be clustered based on the similarity of each word sequence to obtain each clustering result. In a specific implementation, the token sequence can be matched starting from the root node; the token of the current node is compared layer by layer to see if it matches; if the similarity is higher than a set threshold, such as 0.8 or 0.9, which is not limited in this embodiment, then continue downwards; otherwise, a new branch is created; when the leaf node is reached, the log is classified into the corresponding template, and all log data is inserted into the multi-way tree to obtain all clustering results.

[0053] Optionally, in this embodiment, clustering each node based on the similarity of each word sequence may include: merging the first node and the second node when the similarity between the first word sequence in the first node of the multi-branch tree and the second word sequence inserted in the second node of the multi-branch tree meets a set similarity threshold.

[0054] The first node and the second node can be any node in a multi-branch tree; the similarity threshold can be set to be greater than the set similarity threshold, for example, 0.9, 0.95 or 0.98, etc., which are not limited in this embodiment.

[0055] In one example of this embodiment, two log entries are inserted into a multi-way tree. The two log entries are "Log1: Accepted password for alice from 192.168.1.1port 22ssh2" and "Log2: Accepted password for bob from 10.0.0.2port 22ssh2". In the multi-way tree, these can be merged into a single template: "['Accepted','password','for',' <user>','from',' <ip>','port',' <port>','ssh2']".

[0056] Furthermore, the stable portion of the log data in each clustering result can be extracted, and the dynamically changing content can be replaced with the target identifier to obtain the log data pattern set.

[0057] In the specific implementation, each clustering result can be further refined into a template to extract the most stable general log data pattern. In this embodiment, the log data in each clustering result can be compared pairwise using LCS (Longest Common Subsequence) to extract the longest common token subsequence among all logs. For non-common parts, their type (such as IP, timestamp, username, integer, etc.) is determined and replaced with wildcards.

[0058] For example, the IP address can be replaced with <ip>Replace the timestamp with <timestamp>Replace username with <user>Replace the numbers with <int>Replace the string with <string>.

[0059] Step 130: Determine the field structure types included in each log data pattern, extract the variable field content in each field structure type and name the variables to obtain the mapping relationship table between fields and variables.

[0060] Optionally, in this embodiment, after obtaining the log data pattern set, the field structure type contained in each log data pattern can be further determined, and the variable field content in each field structure type can be extracted and named as variables, thereby obtaining a mapping relationship table between fields and variables.

[0061] Optionally, in this embodiment, determining the field structure types included in each log data pattern, extracting the variable field content from each field structure type and naming the variables to obtain a mapping relationship table between fields and variables may include: performing semantic structure parsing on each log data pattern and determining each field structure type based on the parsing results; extracting the variable field content corresponding to each field structure type and assigning temporary variable names to each variable field content; responding to the renaming operation of the target field in each field structure type, determining the target variable name matching the target field, and generating a mapping relationship table between fields and variables based on the target field and the target variable name.

[0062] In an optional implementation of this embodiment, after obtaining each log data pattern, semantic structure parsing can be performed on each log data pattern. Based on the semantic structure parsing results, the structure type of each field contained in the log data pattern can be determined. Further, the variable field content corresponding to each field structure type can be extracted, and temporary variable names can be assigned to each variable field content. When the user performs a renaming operation on the target field, the target variable name matching the target field can be determined, and a mapping relationship table between fields and variables can be generated based on the target field and the target variable name.

[0063] The mapping table can record field name mapping relationships, such as $TEXT1→username, $TEXT2→action, or $INT1→status, etc.

[0064] In the specific implementation, for each log data pattern, the extractable fields, i.e., variable fields, can be identified, and further marked as variables.

[0065] In this embodiment, the field structure type can be CSV, KV, JSON, or XML. In the specific implementation, the extractable fields in the log data pattern can be automatically identified to determine whether they contain CSV fixed delimiter, KV delimiter, JSON / XML brackets, etc. If they contain the above content, these field contents can be extracted and named as variable fields (e.g., IP1, TEXT3), and the field names can be modified according to user interaction to control the field extraction range.

[0066] In an optional implementation of this embodiment, a user interface can be provided that allows users to manually adjust the automatically identified field names, and fields that are not modified are considered not to participate in subsequent extraction.

[0067] In one optional implementation of this embodiment, for each log data pattern, the distribution and pattern of log fields can be learned, including the general timestamp field @timestamp. For example, 10,000 log data can be subjected to line-by-line rule probing to determine whether there are CSV fixed delimiters, KV delimiters, JSON / XML brackets, etc., and corresponding rules are generated. The corresponding fields to be extracted are already highly variable fields, such as $IP1$TEXT3. Each pattern is displayed on the page for users to modify the field names. Fields that are not modified can be considered not to be extracted.

[0068] For example, for CSV, variables can be named according to their position numbers (CSV1, CSV2); for KV, variables can be named by using the key as the field name and extracting and tagging the value (e.g., IP, INT); for JSON / XML, variables can be named according to the parsed structure, extracting key-value pairs and tagging them.

[0069] Step 140: Generate each training sample based on the original log data, the log data pattern set, and the mapping relationship table between the fields and variables.

[0070] The training samples are used to train the field extraction model for log data.

[0071] Optionally, in this embodiment, after completing the above steps, the log information that has been cleaned, pattern mined and field probed can be integrated to generate a structured training sample containing the original log data, log template and field mapping. That is, a training sample can contain a first log data, a log data pattern that matches the first log data and a field mapping relationship that matches the first log data.

[0072] The solution in this embodiment can split log data into individual tokens, and then use the built-in grok regular expression to match these tokens. If the attempt is successful, the token is considered extractable; otherwise, it is written as .*?. For example logs, the field names and values ​​that should be extracted are directly obtained to generate regular expression rules.

[0073] The solution in this embodiment obtains a first log dataset by acquiring raw log data and standardizing each piece of raw log data; it then performs pattern mining on each piece of log data in the first log dataset to obtain a log data pattern set, where each log data pattern corresponds to a log data category; it determines the field structure type contained in each log data pattern, extracts the variable field content from each field structure type and names the variables to obtain a mapping relationship table between fields and variables; it generates training samples based on each piece of raw log data, the log data pattern set, and the mapping relationship table between fields and variables; each training sample is used to train the field extraction model of log data, which can quickly and accurately generate training samples, providing a basis for training the field extraction model of log data and reducing the consumption of human resources.

[0074] Example 2

[0075] Figure 2 This is a flowchart of a training sample generation method according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0076] Step 210: Obtain the raw log data and standardize each piece of raw log data to obtain the first log dataset.

[0077] Step 220: Perform pattern mining on each log data in the first log dataset to obtain a log data pattern set.

[0078] Step 230: Determine the field structure types included in each log data pattern, extract the variable field content in each field structure type and name the variables to obtain the mapping relationship table between fields and variables.

[0079] Step 240: Determine the target log data pattern that matches the target raw log data; determine each entity field in the target raw log data, determine the target field that matches each entity field based on the mapping relationship table between the fields and variables, and determine the field value of the target field; determine the target raw log data, the target log data pattern, the target field, and the field value of the target field as a training sample.

[0080] The target raw log data can be any standardized log data, and this embodiment does not limit it.

[0081] In an optional implementation of this embodiment, after processing each log data using the above steps, a target log data pattern matching the target original log data can be further determined based on the processing results of the above steps; further, each entity field in the target original log data can be determined, and a target field matching each entity field and the field value of the target field can be determined based on the determined mapping relationship table between fields and variables; further, the target original log data, the target log data pattern, the target field, and the field value of the target field can be determined as a training sample.

[0082] The solution in this embodiment can quickly integrate log information after cleaning, pattern mining, and field probing to generate structured training samples containing original log data, log data patterns, and field mappings, providing a basis for training subsequent field extraction models for log data.

[0083] The solution of this invention requires no manual intervention from raw logs to structured training samples, significantly improving log processing efficiency; it effectively extracts common log data patterns, reduces the number of wildcards, and improves template readability; it supports the detection of multiple structure types, automatically names variables, and allows users to interactively modify field names, enhancing flexibility; the generated structured training samples have a unified format and clearly defined fields, making them suitable for supervised learning tasks.

[0084] Example 3

[0085] Figure 3 This is a flowchart of a training method for a field extraction model of log data according to Embodiment 3 of the present invention. This embodiment is applicable to the case of training a field extraction model of log data based on generated training samples. This method can be executed by a training device for the field extraction model of log data. The training device for the field extraction model of log data can be implemented in hardware and / or software, and can be configured in electronic devices such as computers, servers, or tablet computers. Figure 3 As shown, the method includes:

[0086] Step 310: Determine the training samples.

[0087] The training samples are generated by the training sample generation method involved in any of the above embodiments. This embodiment will not repeat the method in detail, as it is not intended to limit this embodiment.

[0088] Step 320: Input the training samples into the target machine learning model for iterative training. Under the condition of stopping the iteration, obtain the field extraction model of the log data.

[0089] The target machine learning model can be a rule-based model, a machine learning model, or a deep learning model; this embodiment does not limit it.

[0090] The iteration stopping condition can be an accuracy condition, for example, stopping iterative training when the calculated accuracy reaches a set threshold (e.g., 0.999); or it can be an iteration count condition, for example, stopping iterative training when the number of iterations reaches a set threshold (e.g., 100,000 times).

[0091] In one optional implementation of this embodiment, after obtaining the training samples, the training samples can be further input into the target machine learning model for iterative training. When the iteration stops, the field extraction model of the log data can be output.

[0092] The solution in this embodiment can train the target machine learning model based on the obtained training samples, and can quickly train the field extraction model of log data, providing a basis for subsequent field extraction of log data.

[0093] Example 4

[0094] Figure 4 This is a flowchart of a log data field extraction method according to Embodiment 4 of the present invention. This embodiment is applicable to situations where log data fields are extracted based on a trained log data field extraction model. This method can be executed by a log data field extraction device, which can be implemented in hardware and / or software. This log data field extraction device can be configured in electronic devices such as computers, servers, or tablet computers. Figure 4 As shown, the method includes:

[0095] Step 410: Obtain target log data.

[0096] The target log data can be any log data obtained. In this embodiment, after obtaining the target log data, it can also be standardized.

[0097] Step 420: Input the target log data into the field extraction model of the log data to obtain the target field that matches the target log and the field value that matches the target field.

[0098] The field extraction model for the log data is trained using the training method for the field extraction model of log data described in any of the above embodiments.

[0099] In this embodiment, after obtaining the target log data of the field to be extracted, the target log data can be further input into the field extraction model of the trained log data to obtain the target field that matches the target log and the field value that matches the target field.

[0100] The solution in this embodiment can quickly determine the target field of the log data and the field value that matches the target field based on the field extraction model of the log data. It can also generate a regular expression rule that matches the target field based on the field extraction model of the log data, providing a basis for subsequent tasks such as anomaly analysis or monitoring of the log data.

[0101] Example 5

[0102] Figure 5 This is a schematic diagram of a training sample generation device according to Embodiment 5 of the present invention, as shown below. Figure 5 As shown, the device includes: an acquisition module 510, a first determination module 520, a second determination module 530, and a third determination module 540.

[0103] The acquisition module 510 is used to acquire raw log data and standardize each piece of raw log data to obtain a first log dataset.

[0104] The first determining module 520 is used to perform pattern mining on each log data in the first log dataset to obtain a log data pattern set; wherein each log data pattern corresponds to a log data category.

[0105] The second determining module 530 is used to determine the field structure type contained in each log data mode, extract the variable field content in each field structure type and name the variables, and obtain a mapping relationship table between fields and variables.

[0106] The third determining module 540 is used to generate training samples based on the original log data, the log data pattern set, and the mapping relationship table between fields and variables; each training sample is used to train the field extraction model of the log data.

[0107] In this embodiment, the scheme involves acquiring raw log data through an acquisition module and standardizing the raw log data to obtain a first log dataset. A first determining module performs pattern mining on each log data in the first log dataset to obtain a log data pattern set, where each log data pattern corresponds to a log data category. A second determining module determines the field structure type of each log data pattern, extracts the variable field content from each field structure type, and names the variables to obtain a mapping table between fields and variables. A third determining module generates training samples based on the raw log data, the log data pattern set, and the mapping table between fields and variables. These training samples are used to train the field extraction model of the log data, enabling rapid and accurate generation of training samples and providing a basis for training the field extraction model of the log data.

[0108] In an optional implementation of this embodiment, the first determining module 520 is specifically used for:

[0109] Based on preset word segmentation symbols, each log data is split into a sequence of words;

[0110] A multi-branch tree is set up, and each of the lexical sequences is inserted into each node of the multi-branch tree. The nodes are clustered based on the similarity of each lexical sequence to obtain the clustering results.

[0111] Extract the stable portion of the log data from each clustering result, and replace the dynamically changing content with the target identifier to obtain the log data pattern set.

[0112] In an optional implementation of this embodiment, the first determining module 520 is further specifically used for:

[0113] If the similarity between the first word sequence in the first node of the multi-branch tree and the second word sequence inserted in the second node of the multi-branch tree meets a set similarity threshold, the first node and the second node are merged.

[0114] In an optional implementation of this embodiment, the second determining module 530 is specifically used for:

[0115] Perform semantic structure parsing on each of the log data patterns, and determine the structure type of each field based on the parsing results;

[0116] Extract the content of the variable fields corresponding to each of the field structure types, and assign temporary variable names to each of the variable field contents;

[0117] In response to the renaming operation of the target field in each of the field structure types, a target variable name matching the target field is determined, and a mapping table between fields and variables is generated based on the target field and the target variable name.

[0118] In an optional implementation of this embodiment, the third determining module 40 is specifically used for:

[0119] Determine the target log data pattern that matches the target raw log data;

[0120] Identify the entity fields in the target raw log data, determine the target fields that match each entity field based on the mapping relationship table between the fields and variables, and determine the field value of the target fields;

[0121] The target raw log data, target log data pattern, target field, and field value of the target field are determined as a training sample.

[0122] The training sample generation apparatus provided in this embodiment of the invention can execute the training sample generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] Example 5

[0124] Figure 6 This is a schematic diagram of the structure of a training device for a field extraction model of log data according to Embodiment Six of the present invention, as shown below. Figure 6 As shown, the device includes a determination module 610 and a training module 620.

[0125] The determining module 610 is used to determine the training sample; the training sample is generated by any training sample generation method in the embodiments of the present invention.

[0126] The training module 620 is used to input the training samples into the target machine learning model for iterative training, and obtain the field extraction model of the log data when the iteration stops.

[0127] In this embodiment, the training samples are determined by a determination module; the training samples are then input into the target machine learning model for iterative training by a training module. When the iteration stops, a field extraction model for log data is obtained. This allows for the rapid training of a field extraction model for log data, providing a basis for subsequent field extraction of log data.

[0128] The training apparatus for the field extraction model of log data provided in the embodiments of the present invention can execute the training method of the field extraction model of log data provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Example 7

[0130] Figure 7 This is a schematic diagram of the structure of a log data field extraction device provided in Embodiment 7 of the present invention. Figure 7 As shown, the device includes an acquisition module 710 and an extraction module 720.

[0131] Among them, the acquisition module 710 is used to acquire target log data;

[0132] The extraction module 720 is used to input the target log data into the field extraction model of the log data to obtain the target field that matches the target log and the field value that matches the target field;

[0133] The field extraction model for the log data is trained using the training method for any field extraction model of log data in this embodiment of the invention.

[0134] In this embodiment, the target log data is acquired by an acquisition module; the target log data is then input into a log data field extraction model by an extraction module to obtain the target field that matches the target log and the field value that matches the target field, providing a basis for subsequent tasks such as anomaly analysis or monitoring of the log data.

[0135] The log data field extraction device provided in this embodiment of the invention can execute the log data field extraction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0136] The collection, storage, use, processing, transmission, provision, and disclosure of log data involved in the technical solutions of this invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0137] Example 8

[0138] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for generating training samples, training a field extraction model for log data, or methods for extracting fields from log data.

[0142] In some embodiments, the method for generating training samples, or the training of a field extraction model for log data, or the method for extracting field data from log data, may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded into and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for generating training samples, or the training of a field extraction model for log data, or the method for extracting field data from log data described above may be performed. Alternatively, in other embodiments, processor 11 may be configured by any other suitable means (e.g., by means of firmware) to execute the method for generating training samples, or the training of a field extraction model for log data, or the method for extracting field data from log data.

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0148] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0149] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0151] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a database detection method as provided in any embodiment of this application.

[0152] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0154] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.< / string> < / int> < / user> < / timestamp> < / ip> < / port> < / ip> < / user>

Claims

1. A method for generating training samples, characterized in that, include: Obtain raw log data and standardize each piece of raw log data to obtain the first log dataset; Based on preset word segmentation symbols, each log data is split into a word sequence; a multi-branch tree is set up, and each word sequence is inserted into each node of the multi-branch tree. Based on the similarity of each word sequence, each node is clustered to obtain each clustering result; the stable part of the log data in each clustering result is extracted, and the dynamically changing content is replaced with the target identifier to obtain the log data pattern set. Each log data pattern corresponds to a log data category; Semantic structure parsing is performed on each of the log data patterns, and the structure type of each field is determined based on the parsing results; the variable field content corresponding to each of the field structure types is extracted, and temporary variable names are assigned to each of the variable field contents; in response to the renaming operation of the target field in each of the field structure types, the target variable name matching the target field is determined, and a mapping relationship table between fields and variables is generated based on the target field and the target variable name; Training samples are generated based on the original log data, the log data pattern set, and the mapping relationship table between fields and variables; each training sample is used to train the field extraction model of the log data.

2. The method for generating training samples according to claim 1, characterized in that, The clustering of the nodes based on the similarity of the word sequences includes: If the similarity between the first word sequence in the first node of the multi-branch tree and the second word sequence inserted in the second node of the multi-branch tree meets a set similarity threshold, the first node and the second node are merged.

3. The method for generating training samples according to claim 1, characterized in that, The generation of training samples based on the original log data, the log data pattern set, and the mapping table between fields and variables includes: Determine the target log data pattern that matches the target raw log data; Identify the entity fields in the target raw log data, determine the target fields that match each entity field based on the mapping relationship table between the fields and variables, and determine the field value of the target fields; The target raw log data, target log data pattern, target field, and field value of the target field are determined as a training sample.

4. A training method for a field extraction model of log data, characterized in that, include: Determine the training samples; The training samples are generated by the training sample generation method according to any one of claims 1-3; The training samples are input into the target machine learning model for iterative training. When the iteration stops, the field extraction model of the log data is obtained.

5. A method for extracting fields from log data, characterized in that, include: Obtain target log data; The target log data is input into the field extraction model of the log data to obtain the target field that matches the target log and the field value that matches the target field; The field extraction model for the log data is trained using the training method for the field extraction model of the log data as described in claim 4.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training sample generation method of any one of claims 1-3, or the training of the log data field extraction model of claim 4, or the log data field extraction method of claim 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for generating training samples according to any one of claims 1-3, or the training of the field extraction model for log data according to claim 4, or the field extraction method for log data according to claim 5.

8. A computer program product comprising a computer program, which, when executed by a processor, implements the method for generating training samples according to any one of claims 1-3, or the training of the field extraction model for log data according to claim 4, or the field extraction method for log data according to claim 5.

Images