Log analysis method, classification model training method, device and equipment

Abstract

The invention discloses a log analysis method, a classification model training method, a device and equipment. The method comprises the following steps: acquiring a plurality of first logs belonging to different fields; inputting the plurality of first logs into a classification model, and outputting a domain identifier of a domain to which each first log in the plurality of first logs belongs, the classification model being obtained by training according to a reference number of second logs and the domain identifiers corresponding to the second logs; and determining a universal first log analysis template of the plurality of first logs according to each domain identifier and the log set corresponding to each domain identifier. According to the method and the device, the domain identifier of the domain to which each first log in the plurality of first logs belongs can be distinguished through the classification model, and the first logs are divided according to the domain identifiers to obtain the log sets corresponding to the domain identifiers; and then, according to the log set corresponding to each domain identifier, determining a plurality of analysis templates (namely, the first log analysis templates) universal for the first logs, so that the purpose of log analysis of multiple domains is achieved, and the universality is relatively high.

Description

Technical Field

[0001] The present application relates to the field of log parsing technology, and in particular to a log parsing method, a classification model training method, a device, and equipment. Background Art

[0002] The integration of network and device systems has been widely used in various fields, including finance, healthcare, transportation, and communications. However, device systems are prone to failures, resulting in device malfunctions. To identify device system failures, one related technology records device system information in logs, then parses the logs to generate log parsing results, which are used to indicate device system failures. However, in multi-domain scenarios, parsing logs from multiple fields is affected by the diversity of log formats, resulting in poorly accurate log parsing results. Summary of the Invention

[0003] This application provides a log parsing method, a classification model training method, an apparatus, and a device to solve the problems provided by related technologies. The technical solutions are as follows:

[0004] In a first aspect, a log parsing method is provided, the method comprising: obtaining a plurality of first logs, the plurality of first logs belonging to different fields; inputting the plurality of first logs into a classification model, and outputting a domain identifier of a domain to which each of the plurality of first logs belongs, the classification model being trained based on a reference number of second logs and the domain identifiers corresponding to the second logs; determining a first log parsing template based on a second log parsing template corresponding to each of the domain identifiers, the first log parsing template being used to characterize the parsing results of the plurality of first logs in a plurality of fields, and the second log parsing template being used to characterize the parsing results of the plurality of first logs in a single field.

[0005] In this application, a classification model is used to classify multiple first logs to obtain the domain identifier of the domain to which each first log belongs, and the first log parsing templates of multiple first logs in multiple fields are determined based on the second log parsing templates corresponding to each domain identifier. It can be seen that this application converts the parsing problem of first logs in multiple fields into the parsing problem of first logs in multiple single domains, so that the parsing results obtained by parsing on each single domain are more accurate.

[0006] In one possible implementation, before determining the first log parsing template, the method further includes: determining a second log parsing template corresponding to the domain identifier based on the domain identifier and the first log corresponding to the domain identifier; the determining of the first log parsing template includes: combining the second log parsing templates corresponding to each of the domain identifiers to obtain the first log parsing template. In the present application, based on the domain identifier and the set of logs corresponding to the domain identifier, the log parsing template corresponding to each domain identifier is extracted. In this way, extracting the log parsing template within a single domain will more fully explore the domain semantic features of the log and obtain higher template quality.

[0007] In one possible implementation, determining the second log parsing template corresponding to the domain identifier includes: extracting first key information and second key information of the first log corresponding to the domain identifier, the first key information being used to characterize attribute information of the domain to which the domain identifier belongs, and the second key information being used to characterize parameter values ​​corresponding to the attribute information; extracting key features from the first key information and the second key information, the key features being features with the highest frequency of occurrence; and determining the second log parsing template corresponding to the domain identifier based on the key features.

[0008] In one possible implementation, before inputting the multiple first logs into the classification model and outputting the domain identifier of the domain to which each first log in the multiple first logs belongs, the method further includes: obtaining a reference number of second logs and the domain identifier corresponding to the second logs; using the reference number of second logs as input data, and using the domain identifier corresponding to each second log in the reference number of second logs as output data, to train the classification model. In the present application, the classification model can be trained on logs from multiple different source fields. The classification model can transfer domain-specific semantic knowledge without supervision in most multi-source log sets. It greatly reduces the workload of manual annotation through unsupervised transfer learning, and further reduces the implementation cost of log processing.

[0009] In one possible implementation, obtaining a reference number of second logs includes obtaining the reference number of third logs; and if the third logs include invalid information, deleting the invalid information from the third logs to obtain the second logs, where the invalid information is information in the third logs that does not contribute to the generation of the first log parsing template. In this application, by deleting specific symbols contained in the logs, the generalization capability of the classification model can be effectively improved, thereby enhancing the versatility of the classification model.

[0010] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0011] In one possible implementation, obtaining a reference number of second logs includes obtaining the reference number of third logs; and if the third logs include irregular characters, correcting the irregular characters in the third logs using regular expressions to obtain the second logs. In the present application, by correcting the irregular characters in the third logs using regular expressions, the information in the second logs used to train the classification model is made more concise, effectively eliminating interfering information, and thereby making the classification model trained using the second logs more accurate.

[0012] In one possible implementation, obtaining a reference number of second logs includes obtaining the reference number of third logs; and if the third logs include phrases containing special characters, splitting the phrases in the third logs and removing the special characters from the phrases to obtain the second logs. In the present application, by splitting the phrases and removing the special characters from the phrases, the processed phrases are more readable, facilitating the provision of accurate training data for subsequent analysis model training, thereby improving the accuracy of model training.

[0013] On the second aspect, a method for training a classification model is provided, the method comprising: obtaining a reference number of second logs and domain identifiers corresponding to the second logs; using the reference number of second logs as input data, and using the domain identifier corresponding to each second log in the reference number of second logs as output data, to train a classification model, the classification model being used to distinguish the domain identifier of the domain to which each first log belongs in a plurality of first logs, the plurality of first logs belonging to different fields. In the present application, the classification model can be trained on logs from a plurality of different source fields. The classification model can transfer domain-specific semantic knowledge without supervision in most multi-source log sets, and it greatly reduces the workload of manual annotation through unsupervised transfer learning, further reducing the implementation cost of log processing.

[0014] In addition, in this application, by using a reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data, a classification model is trained. The resulting classification model can accurately identify the domain to which the log belongs and remains robust to domain classification errors of the log. Based on this classification model, the foundation is laid for the subsequent generation of log parsing templates for multi-domain logs, which significantly improves the accuracy and efficiency of subsequent log parsing.

[0015] In one possible implementation, obtaining a reference number of second logs includes: obtaining the reference number of third logs; and if the third log includes invalid information, deleting the invalid information from the third log to obtain the second log, wherein the invalid information is information in the third log that does not contribute to the generation of a first log parsing template, the first log parsing template being used to represent the parsing results of multiple first logs in multiple fields. In the present application, by deleting specific symbols contained in the logs, the generalization ability of the classification model can be effectively improved, thereby increasing the versatility of the classification model.

[0016] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0017] In one possible implementation, obtaining a reference number of second logs includes obtaining the reference number of third logs; and if the third logs include irregular characters, correcting the irregular characters in the third logs using regular expressions to obtain the second logs. In the present application, by correcting the irregular characters in the third logs using regular expressions, the information in the second logs used to train the classification model is made more concise, effectively eliminating interfering information, and thereby making the classification model trained using the second logs more accurate.

[0018] In one possible implementation, obtaining a reference number of second logs includes obtaining the reference number of third logs; and if the third logs include phrases containing special characters, splitting the phrases in the third logs and removing the special characters from the phrases to obtain the second logs. In the present application, by splitting the phrases and removing the special characters from the phrases, the processed phrases are more readable, facilitating the provision of accurate training data for subsequent analysis model training, thereby improving the accuracy of model training.

[0019] In a third aspect, a log parsing device is provided, comprising: an acquisition module for acquiring a plurality of first logs, wherein the plurality of first logs belong to different fields; an output module for inputting the plurality of first logs into a classification model and outputting a domain identifier of the domain to which each of the plurality of first logs belongs, wherein the classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs; a determination module for determining a first log parsing template based on a second log parsing template corresponding to each of the domain identifiers, wherein the first log parsing template is used to represent the parsing results of the plurality of first logs in a plurality of fields, and the second log parsing template is used to represent the parsing results of the plurality of first logs in a single field.

[0020] In a possible implementation, the determination module is used to: determine the second log parsing template corresponding to the domain identifier based on the domain identifier and the first log corresponding to the domain identifier; and combine the second log parsing templates corresponding to each of the domain identifiers to obtain the first log parsing template.

[0021] In one possible implementation, the determination module is used to: extract first key information and second key information of the first log corresponding to the domain identifier, the first key information is used to characterize attribute information of the domain to which the domain identifier belongs, and the second key information is used to characterize parameter values ​​corresponding to the attribute information; extract key features from the first key information and the second key information, the key features being the features with the highest frequency of occurrence; and determine the second log parsing template corresponding to the domain identifier based on the key features.

[0022] In one possible implementation, the apparatus further includes a training module; the acquisition module is configured to acquire a reference number of second logs and domain identifiers corresponding to the second logs. The training module is configured to train the classification model using the reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data.

[0023] In one possible implementation, the acquisition module is used to: obtain the reference number of third logs; if the third log includes invalid information, delete the invalid information in the third log to obtain the second log, where the invalid information is information in the third log that does not contribute to the generation of the first log parsing template.

[0024] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0025] In a possible implementation, the acquisition module is configured to: acquire the reference number of third logs; and, if the third logs include irregular characters, use regular expressions to correct the irregular characters in the third logs to obtain the second logs.

[0026] In a possible implementation, the acquisition module is configured to: acquire the reference number of third logs; and if the third logs include phrases containing special characters, split the phrases in the third logs and remove the special characters in the phrases to obtain the second logs.

[0027] In a fourth aspect, a training device for a classification model is provided, the device comprising: an acquisition module for acquiring a reference number of second logs and domain identifiers corresponding to the second logs; a training module for taking the reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data to train a classification model, the classification model being used to distinguish the domain identifier of the domain to which each first log in multiple first logs belongs, the multiple first logs belonging to different fields.

[0028] In one possible implementation, the acquisition module is used to: obtain the reference number of third logs; if the third log includes invalid information, delete the invalid information in the third log to obtain the second log, where the invalid information is information in the third log that does not contribute to the generation of a first log parsing template, and the first log parsing template is used to characterize the parsing results of multiple first logs in multiple fields.

[0029] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0030] In a possible implementation, the acquisition module is configured to: acquire the reference number of third logs; and, if the third logs include irregular characters, use regular expressions to correct the irregular characters in the third logs to obtain the second logs.

[0031] In a possible implementation, the acquisition module is configured to: acquire the reference number of third logs; and if the third logs include phrases containing special characters, split the phrases in the third logs and remove the special characters in the phrases to obtain the second logs.

[0032] In a fifth aspect, an electronic device is provided, comprising a memory and a processor; at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to enable the electronic device to implement the method described in the first aspect, or to implement the method described in the second aspect.

[0033] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method as described in the first aspect, or to implement the method as described in the second aspect.

[0034] In the seventh aspect, a computer program (product) is provided, which includes: computer program code, which, when executed by a computer, enables the computer to execute the method described in the first aspect or the method described in the second aspect.

[0035] In an eighth aspect, a computer-readable storage medium is provided, which stores a program or instruction. When the program or instruction runs on a computer, the method described in the first aspect or the method described in the second aspect is executed.

[0036] In the ninth aspect, a chip is provided, comprising a processor for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes the method described in the first aspect or the method described in the second aspect.

[0037] In the tenth aspect, another chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method described in the first aspect or the method described in the second aspect.

[0038] It should be understood that the beneficial effects achieved by the technical solutions of the third to tenth aspects of this application and the corresponding possible implementation methods can be referred to the technical effects of the first and second aspects and their corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 2 A structural block diagram of another electronic device provided in an embodiment of the present application;

[0041] Figure 3 A flow chart of a classification model training method provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a classification model training method provided in an embodiment of the present application during the classification model training process;

[0043] Figure 5 A flow chart of a log parsing method provided in an embodiment of the present application;

[0044] Figure 6 (a) is a schematic diagram showing a comparison between a classification result output by a classification model provided in an embodiment of the present application and an actual classification result;

[0045] Figure 6 (b) is a schematic diagram showing a comparison between a classification result output by a classification model provided in an embodiment of the present application and an actual classification result;

[0046] Figure 7 A schematic diagram comparing the running time of a log parsing algorithm provided in an embodiment of the present application with the running time of a log parsing algorithm in related art;

[0047] Figure 8 A structural block diagram of a log parsing device provided in an embodiment of the present application;

[0048] Figure 9 A structural block diagram of a classification model training device provided in an embodiment of the present application;

[0049] Figure 10 This is a structural block diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0051] The integration of network and device systems has been widely adopted in various fields, including finance, healthcare, transportation, and communications. However, these systems are prone to failures, which can cause equipment to malfunction. To identify system failures, one related technique involves recording system information in logs, which are then manually analyzed by personnel for potential faults. However, due to the large volume of data in the logs, manual log analysis is time-consuming and requires extensive knowledge of operations and maintenance.

[0052] Therefore, log parsing emerged. Log parsing plays a crucial role in log analysis. It converts unstructured logs into structured logs, providing an accurate data foundation for subsequent log analysis. This allows personnel to focus on problem solving rather than rummaging through reams of log data to identify faults. However, in multi-domain scenarios, the log formats of each domain vary. Log parsing is hampered by this diversity and cannot parse logs from multiple domains.

[0053] To address the above technical issues, embodiments of the present application provide a log parsing method that can be applied to an electronic device. The method may include: the electronic device obtaining multiple first logs belonging to different domains. The electronic device inputs the multiple first logs into a classification model and outputs a domain identifier for the domain to which each of the multiple first logs belongs. The classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs. The electronic device determines a first log parsing template for the multiple first logs based on the domain identifiers and the log sets corresponding to each domain identifier. The first log parsing template is a common parsing template for the multiple first logs. In the present application, the electronic device can use the classification model to distinguish the domain identifiers of the domains to which each of the multiple first logs belongs. The electronic device divides the first logs according to the domain identifiers to obtain log sets corresponding to the domain identifiers. The electronic device then determines a common parsing template (i.e., a first log parsing template) for the multiple first logs based on the log sets corresponding to the domain identifiers, thereby achieving the purpose of parsing logs in multiple domains and having high versatility. In addition, the parsing of logs in multiple domains does not require human intervention, resulting in high log parsing efficiency and time savings.

[0054] In some embodiments, the electronic device determines the first log parsing template of multiple first logs based on each domain identifier and the log set corresponding to each domain identifier. This can be implemented as follows: the electronic device determines the second log parsing template corresponding to each domain identifier based on each domain identifier and the log set corresponding to each domain identifier. The electronic device determines the first log parsing template based on each second log parsing template. It should be understood that the electronic device divides the multiple first logs into log sets corresponding to the domain identifiers according to the domain identifiers, and determines the second log parsing template corresponding to each domain identifier. The electronic device then combines these second log parsing templates into the first log parsing template. It can be seen that in the present application, the first logs of multiple fields are converted into log sets of a single field, and the second log solution templates corresponding to each field are obtained. Finally, the second log parsing templates of each field are combined into a universal log parsing template (i.e., the first log parsing template). This means that the log parsing method of the present application converts the log parsing problem of multiple fields into the log parsing problem of a single field, and improves the accuracy of the log parsing of multiple fields through detailed parsing of each field.

[0055] In some embodiments, before the classification model outputs the domain identifier of the domain to which each first log in the plurality of first logs belongs, the log parsing method provided by the embodiment of the present application may further include: the electronic device trains the classification model based on a reference number of second logs and the domain identifiers corresponding to the second logs. Figure 1 This is a structural block diagram of the above electronic equipment. Figure 1As shown, the electronic device 100 may include: a domain perception module. The domain perception module is used to train a classification model based on a reference number of historical logs (i.e., second logs) and domain identifiers corresponding to the historical logs. In one example, the working principle of the electronic device 100 may be: Step ① The domain perception module of the electronic device 100 identifies the domain to which the second logs of the parameter number belong, and obtains the corresponding relationship between each second log and the domain identifier to which it belongs, such as Figure 1 Here, H represents the second log, and x represents the domain identifier. In step 2, the domain-aware module compares the domain identifier corresponding to each of these second logs with the theoretical results to obtain an error result. This error result is used to indicate the consistency between the classification result and the theoretical result. After multiple iterations, when the error result obtained by the electronic device is minimized (e.g., the error result is 0), classification model training is complete and the electronic device has obtained a classification model.

[0056] After the electronic device obtains the classification model, the electronic device uses the classification model to classify and parse the multi-source logs. Figure 2 It is another structural block diagram of the above-mentioned electronic device. The difference compared to the above is that the electronic device 100 can also include: a log parsing module. The domain perception module is electrically connected to the log parsing module. The domain perception module is also used to input multiple logs to be parsed (i.e., the first log) into the classification model, and output the domain identifier of the domain to which the log belongs. The log parsing module is used to extract the log parsing template corresponding to each field from the log set of the field (i.e., the second log parsing template), and merge the log parsing templates of each field to obtain a parsing template common to multiple fields (i.e., the first log parsing template). In one example, the working principle of the electronic device 100 can be: Figure 1 In step ①, the domain perception module of the electronic device 100 determines the domain identifier corresponding to each of the historical logs based on the reference number of historical logs (i.e., the second log). In step ②, the domain perception module trains the domain perception layer based on the reference number of historical logs and the domain identifiers corresponding to the historical logs, and obtains a classification model. Figure 2 In step ③ and step ④, the electronic device inputs multiple logs to be parsed (i.e., the first log) into the classification model and outputs the domain identifier of the field to which each log to be parsed belongs. In step ⑤, the log parsing module of the electronic device divides the multiple logs to be parsed into log sets according to the domain identifier (e.g., Figure 2 As shown in the log set 1 (logs(D1)), log set 2 (logs(D2)) ... log set n (logs(Dn))). Step 6 The log parsing module extracts the key information of each log set and obtains the log parsing template corresponding to each log set. For example, the log parsing module extracts the following Figure 2 The key information of log set 1 (logs(D1)) shown in the figure is obtained, and the corresponding log set 1 (logs(D1)) is obtained. Figure 2 The log parsing template 1 (templates(D1)) is shown in Figure 1. Similarly, the log parsing module extracts Figure 2 The key information of log set 2 (logs(D2)) shown in the figure is obtained, and the corresponding log set 2 (logs(D2)) is obtained. Figure 2 The log parsing template 2 (templates(D2)) is shown in Figure 2. Similarly, the log parsing module extracts Figure 2 The key information of the log set n (logs (Dn)) shown in the figure is obtained, and the corresponding log set n (logs (Dn)) is obtained. Figure 2 The log parsing template n (templates(Dn)) shown in FIG. 7 : The log parsing template merges the log parsing templates corresponding to the domain identifiers into a common log parsing template (ie, the first log parsing template).

[0057] As can be seen, the log parsing method provided in the embodiments of the present application can be divided into two stages. The first stage is the training stage of the classification model. The second stage is the application stage of the classification model. The following describes each stage in detail.

[0058] The first stage is the training stage of the classification model.

[0059] Figure 3 A flow chart of a training method for a classification model provided in an embodiment of the present application. Figure 3 As shown, the method can be applied to electronic devices. The method may include: S301-S302.

[0060] S301: The electronic device obtains a reference number of second logs and domain identifiers corresponding to the second logs.

[0061] The reference number may be a predetermined number, such as 100 or 1000, but is not specifically limited in the embodiments of the present application.

[0062] The second log can be understood as a historical log. The domain identifier corresponding to the second log is determined by the domain to which the log belongs.

[0063] For example, the fields to which the multiple second logs belong may include: Blue Gene / L supercomputer (BGL), high performance cluster (HPC), Hadoop distributed file system (HDFS), ZooKeeper service (Zookeeper), network tool (proxifier software, proxifier), Linux operating system, Windows operating system, Hadoop mapreduce job (Hadoop), cloud platform management project (OpenStack infrastructure, OpenStack), big data processing framework Apache Spark, MacOS operating system, Android operating system, health app (health app), Apache web server software (Apache), independent daemon process OpenSSH and email client (Thunderbird supercomputer, Thunderbird), etc. They are not listed one by one in the embodiments of the present application.

[0064] Correspondingly, the domain identifier corresponding to the second log is determined according to the field to which the second log belongs, and is not specifically limited in the embodiments of the present application.

[0065] In a possible implementation, S301 may include S3011 and S3012.

[0066] S3011. The electronic device obtains a reference number of third logs.

[0067] S3012: When the third log includes invalid information, the electronic device deletes the invalid information in the third log to obtain a second log.

[0068] Invalid information can be understood as information in the third log that does not contribute to generating the log parsing template. This non-contributing information can be understood as information in the third log that is not included in the log parsing template. In other words, invalid information is not key information generated by the log parsing template.

[0069] In one example, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information. For example, the invalid information may include time information, identification information, and a specific symbol. Alternatively, the invalid information may include time information and a specific symbol. Alternatively, the invalid information may include log header information, such as "combo sshd (pam_unix)." Alternatively, the invalid information may include a specific symbol, such as "R24-M0-NE-C." Of course, other scenarios for invalid information are possible, which are not listed in the embodiments of this application.

[0070] Continuing with the above example, if the invalid information in the third log includes log header information, such as "combo sshd(pam_unix)", the electronic device deletes the log header information from the third log to obtain the second log. In this way, by deleting highly similar headers in logs from various fields, the present embodiment effectively prevents overfitting during subsequent classification model training.

[0071] Continuing with the above example, if the invalid information in the third log includes specific symbols, such as "R24-M0-NE-C," the electronic device will delete the specific symbols from the third log. These specific symbols may cause the classification model to overfit to the simple formatting features of the log header, rather than effectively classifying the logs based on domain-specific semantic content. Consequently, the generalization capability of the classification model may be affected. Thus, in the embodiments of the present application, by deleting the specific symbols contained in the logs, the generalization capability of the classification model can be effectively improved, thereby enhancing its versatility.

[0072] In another possible implementation, S301 may further include S3013.

[0073] S3013: When the third log includes irregular characters, the electronic device uses a regular expression to correct the irregular characters in the third log to obtain a second log.

[0074] Irregular characters can be understood as character strings such as internet protocol addresses (IP addresses) and file storage paths.

[0075] Regular expressions can be used to find, replace, extract, and verify specific patterns in logs. For example, a regular expression might include: [IP] to match IP addresses in logs. For example, a regular expression might include: [FILE] to match file storage paths and uniform resource locators (URLs) in logs. For example, a regular expression might include: [NUM] to match other numeric sequences in logs.

[0076] In the embodiment of the present application, regular expressions are used to correct irregular characters in the third log, so that the information in the second log used to train the classification model is more concise, and interference information is effectively discarded, thereby making the classification model trained using the second log more accurate.

[0077] In yet another possible implementation, S301 may further include S3014.

[0078] S3014: When the third log includes a phrase containing special characters, the electronic device splits the phrase in the third log and removes the special characters in the phrase to obtain a second log.

[0079] A phrase containing special characters can be understood as a phrase containing special characters. For example, a phrase contains a connector, such as "set_timer_for_scene". For example, a phrase contains capital letters, such as "SET TIMER FORSCENE". For example, a phrase contains two consecutive words, such as "SETTIMER".

[0080] S3014 can be implemented as follows: the electronic device analyzes the semantics of the phrases in the third log, divides the phrases based on the semantics, and removes special characters from the phrases to obtain the second log. For example, the electronic device splits "SETTIMER_FOR_SCENE" to obtain "SET TIMER_FOR_SCENE", then removes special characters from "SET TIMER_FOR_SCENE" and converts uppercase letters to lowercase to obtain "set timer for scene".

[0081] In the embodiment of the present application, by splitting the phrases and removing special characters in the phrases, the processed phrases are more readable, which facilitates providing accurate training data for subsequent analysis model training and improves the accuracy of model training.

[0082] In some embodiments, the above steps S3012-S3014 may be implemented in parallel or in a progressive manner, such as sequentially implementing S3012-S3014, which is not specifically limited in the embodiments of the present application. In order to obtain a more accurate second log, in a specific implementation, the electronic device executes S3012-S3014 sequentially.

[0083] In a possible implementation, the implementation of S301 can be found in Figure 1 For details, see step ① shown in Figure 1 The relevant descriptions in step ① shown in the figure will not be repeated here.

[0084] S302: The electronic device uses a reference number of second logs as input data and uses a domain identifier corresponding to each second log in the reference number of second logs as output data to train a classification model.

[0085] The classification model can use a pre-trained model, such as the BERT (bidirectional encoder representations from transformer) model. The implementation of S302 can refer to the training method of the BERT model, which will not be repeated here. Of course, the classification model can also use other models, which are not specifically limited in the embodiments of this application.

[0086] In a possible implementation, the implementation of S302 can be found in Figure 1 For details, see step ② shown in Figure 1 The relevant description in step ② shown in the figure will not be repeated here.

[0087] In another possible implementation, S302 can be implemented as follows: the electronic device obtains a classification result based on a reference number of second logs, and the classification result is the domain identifier corresponding to each second log in the reference number of second logs. The electronic device compares the domain identifier corresponding to each second log in these second logs with the theoretical result to obtain an error result, and the error result is used to characterize the consistency between the classification result and the theoretical result. The larger the numerical value of the error result, the smaller the consistency between the classification result and the theoretical result, and the smaller the numerical value of the error result, the greater the consistency between the classification result and the theoretical result. After multiple iterations, when the numerical value of the error result obtained by the electronic device is the smallest (such as the numerical value of the error result is 0), it means that the classification model training is completed and the electronic device obtains the classification model.

[0088] Exemplarily, it is assumed that the fields to which the reference number of second logs belong may include BGL, HPC, HDFS, zookeeper, proxifier, Linux, Windows, Hadoop, OpenStack, Apache Spark, MacOS, Android, health app, Apache, OpenSSH, and Thunderbird.

[0089] In the first training cycle (such as Figure 4 ), the electronic device obtains classification result 1 based on the reference number of second logs, and the classification result 1 is the domain identifier 1 corresponding to each second log in the reference number of second logs. The electronic device compares the domain identifier 1 corresponding to each second log with the theoretical result to obtain an error result 1. The theoretical result is the standard domain identifier corresponding to each second log in the reference number of second logs. Figure 4As shown, when the value of the error result 1 is 0.5, it means that the classification result 1 is quite different from the theoretical result. At this time, the electronic device continues to train. In the second training cycle (such as Figure 4 (shown as "2" on the X-axis in the figure), the electronic device obtains classification result 2 based on the reference number of second logs and the domain identifier in classification result 1 that is consistent with the theoretical result. Classification result 2 is domain identifier 2 corresponding to each second log in the reference number of second logs. The electronic device compares domain identifier 2 corresponding to each second log with the theoretical result to obtain error result 2. Figure 4 As shown in , when the value of error result 2 is 0.05, it means that there is still a difference between classification result 1 and theoretical result. And so on, until the error result n obtained by the electronic device, such as Figure 4 In the example, n is 7. At this point, the error result 7 is close to 0.0, indicating that the classification result 7 is substantially consistent with the theoretical result. The electronic device can then determine that the classification model training is complete, meaning that the electronic device has obtained the classification model.

[0090] Then, the classification model obtained by the electronic device through the above training is applied to the second log of the above reference number, and the classification results of each field as shown in Table 1 can be obtained. As shown in Table 1:

[0091] Table 1

[0092] domain precision recall F1-score BGL 1.0000 1.0000 1.0000 HPC 1.0000 1.0000 1.0000 HDFS 1.0000 0.9960 0.9980 zookeeper 1.0000 1.0000 1.0000 proxifier 1.0000 0.9940 0.9970 Linux 0.9714 0.9315 0.9510 Hadoop 0.9298 0.9464 0.9381 OpenStack 1.0000 1.0000 1.0000 Spark 0.9038 0.9592 0.9307 MacOS 0.8939 0.9516 0.9219 Android 0.9940 0.9940 0.9940 health app 1.0000 1.0000 1.0000 apache 1.0000 0.7500 0.8571 OpenSSH 0.9982 1.0000 0.9991 Windows 0.9960 0.9940 0.9950 Thunderbird 0.9939 0.9840 0.9889

[0093] In Table 1, the accuracy can be expressed as: Precision = TP / (TP + FP); the recall can be expressed as: Recall = TP / (TP + FN); and the score is the average of the accuracy and recall, that is, the score can be expressed as: F1-score = (TP / (TP + FP) + TP / (TP + FN)) / 2. TP represents the sum of log pairs x and y that are manually classified into the same cluster and have the same prediction template, with x and y being two randomly selected logs; TN represents the sum of log pairs x and y that are manually classified into different clusters and have different prediction templates; FP represents the sum of log pairs x and y that are manually classified into different clusters but have the same prediction template; and FN represents the sum of log pairs x and y that are manually classified into the same cluster but have different prediction templates.

[0094] It can be seen that the classification results output by the classification model obtained through the above training have high accuracy, recall rate and score. Therefore, the classification model obtained in the embodiment of the present application can accurately classify multi-source logs. Therefore, in the embodiment of the present application, by taking the reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data, the classification model is trained. The obtained classification model can accurately identify the field to which the log belongs and maintain robustness to the field classification errors of the log. Based on this classification model, the foundation is laid for the subsequent generation of log parsing templates for multi-field logs, so that the accuracy and parsing efficiency of subsequent log parsing are significantly improved.

[0095] The second stage is the application stage of the classification model.

[0096] Figure 5 A flow chart of a log parsing method provided in an embodiment of the present application. Figure 5 As shown, the method can be applied to electronic devices, and the method includes: S501-S503.

[0097] S501: The electronic device obtains a plurality of first logs, where the plurality of first logs belong to different fields.

[0098] It should be understood that the multiple first logs are logs in multiple fields, or called multi-source logs.

[0099] In a possible implementation, the implementation of S501 can be found in Figure 2 For details, see step ③ shown in Figure 2 The relevant description in step ③ shown in the figure will not be repeated here.

[0100] S502: The electronic device inputs a plurality of first logs into a classification model, and outputs a domain identifier of a domain to which each of the plurality of first logs belongs. The classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs.

[0101] In a possible implementation, the implementation of S502 can be found in Figure 2 For details, see step ④ shown in Figure 2 The relevant description in step ④ shown in is not repeated here.

[0102] The training of the classification model can be found in the relevant description in the first stage above and will not be repeated here.

[0103] For example, the domains to which the multiple first logs belong may include BGL, HPC, HDFS, Zookeeper, Proxifier, Linux, and Hadoop. Then, in S502, the electronic device inputs these first logs into a classification model to obtain the domain identifier of each domain to which these first logs belong. Using the above classification model, the evaluation results of the classification model on the multiple first logs are obtained, as shown in Table 2.

[0104] Table 2

[0105] domain precision recall F1-score BGL 1.0000 1.0000 1.0000 HPC 1.0000 1.0000 1.0000 HDFS 1.0000 0.9960 0.9980 zookeeper 1.0000 1.0000 1.0000 proxifier 1.0000 0.9940 0.9970 Linux 0.9714 0.9315 0.9510 Hadoop 0.9298 0.9464 0.9381

[0106] It can be seen that the classification results output by the classification model obtained through the above training have high accuracy, recall rate and score. Therefore, the classification model obtained in the embodiment of the present application can accurately classify multi-source logs. Therefore, in the embodiment of the present application, by inputting multiple first logs into the classification model, the domain identifier corresponding to each first log in the multiple first logs is obtained. It can be seen that the classification model can accurately identify the fields to which multiple first logs belong, laying the foundation for the subsequent generation of log parsing templates for multi-field logs, so that the accuracy and parsing efficiency of subsequent log parsing are significantly improved.

[0107] S503: The electronic device determines a first log parsing template according to each domain identifier and the second log parsing template corresponding to each domain identifier.

[0108] The first log parsing template can be used to represent the parsing results of multiple first logs in multiple fields. The first log parsing template can be understood as a collection of multiple log parsing templates.

[0109] It should be understood that the electronic device can cluster multiple first logs based on each domain identifier to obtain a log set consisting of multiple first logs corresponding to each domain identifier. Based on the log set corresponding to each domain identifier, the electronic device obtains a second log parsing template corresponding to each domain identifier. Based on these second log parsing templates, the electronic device obtains a first log parsing template. The first log parsing template can be understood as the set of log parsing templates corresponding to these first logs.

[0110] In a possible implementation, the implementation of S503 can be found in Figure 2 For details on steps ⑤, ⑥, and ⑦ shown in Figure 2 The relevant descriptions in step ⑤, step ⑥ and step ⑦ shown in are not repeated here.

[0111] In another possible implementation, S503 may include S5031 and S5032.

[0112] S5031: The electronic device determines a second log parsing template corresponding to each domain identifier based on each domain identifier and the first log corresponding to each domain identifier. S5031 may include S50311, S50312, and S50313.

[0113] S50311. The electronic device divides the multiple first logs according to the domain identifiers to obtain a log set consisting of the multiple first logs corresponding to the domain identifiers.

[0114] Exemplarily, the plurality of first logs may include log 1, log 2, log 3, log 4, log 5, log 6, log 7, log 8, log 9, and log 10. Log 1 corresponds to identifier 1, log 2 corresponds to identifier 5, log 3 corresponds to identifier 2, log 4 corresponds to identifier 1, log 5 corresponds to identifier 5, log 6 corresponds to identifier 2, log 7 corresponds to identifier 1, log 8 corresponds to identifier 5, log 9 corresponds to identifier 1, and log 10 corresponds to identifier 2. Thus, the electronic device can obtain: identifier 1 corresponds to log set 1, which includes log 1, log 4, log 7, and log 9; identifier 2 corresponds to log set 2, which includes log 3, log 6, and log 10; identifier 5 corresponds to log set 3, which includes log 2, log 5, and log 8.

[0115] S50312. The electronic device extracts first key information and second key information from each log set, where the first key information is used to represent attribute information of the domain to which the domain identifier belongs, and the second key information is used to represent a parameter value corresponding to the attribute information.

[0116] It can be understood that the first key information can be understood as fixed information, and the second key information can be understood as variable information. For example, if the first key information is the voltage of device 1, then the second key information can be the voltage value of device 1. For example, if the first key information is the temperature of device 2, then the second key information can be the temperature value of device 2. For example, if the first key information is the vehicle, then the second key information can be the vehicle number.

[0117] S50313: The electronic device determines a second log parsing template corresponding to each domain identifier based on the first key information and the second key information.

[0118] In one possible implementation, S50313 may be implemented as follows: the electronic device extracts key features from the first key information and the second key information, where the key features are the features with the highest frequency of occurrence, and determines, based on the key features, a second log parsing template corresponding to each domain identifier.

[0119] For example, suppose a first log entry records the arrival of Train A at Station A at 9:00 AM. Another first log entry records the arrival of Train B at Station A at 10:00 AM. The most frequently occurring features are "_ train," "at," "_:_," and "arrived at _ station." The electronic device then assembles these features into a second log parsing template, which may include "_ train arrived at _:_ at _ station." Other examples are possible and are not listed here.

[0120] The second log parsing template can be understood as multiple templates. That is, the second log parsing template can be multiple log parsing templates corresponding to each domain identifier. In other words, the second log parsing template is a log parsing template set corresponding to each domain identifier.

[0121] S5032: The electronic device combines the second log parsing templates into a first log parsing template.

[0122] It should be understood that the electronic device combines the second log parsing templates to obtain the first log parsing template.

[0123] For example, assuming that a data set S' includes m first logs, the expression can be S'={X1', X2', ..., X m '}, where these first logs come from L' fields. These first logs can be divided into L' groups of log sets. Then, the composition of a group of log sets can be expressed as: q i ′ represents a log X i ’ corresponding field.

[0124] Afterwards, the electronic device uses the single-domain log parsing algorithm f(·) to train a log template set for each set of logs, and generates a unified first log parsing template T from these log template sets, which can be expressed as:

[0125] T=T1·T2·…·T L′ =f(S′1)·f(S′2)·…·f(S′ L′ )

[0126] Among them, T represents the first log parsing template, L' represents the number of fields of multi-source logs, T L’ It is represented as the second log parsing template corresponding to the L' field; f(S' L′ ) represents the log parsing algorithm corresponding to the domain L'. S'1 represents the log set from domain l.

[0127] In practical applications, following the above example, log parsing methods in related technologies are adopted, such as FT-tree (frequent template tree)

[21] , log data clustering algorithm IPLoM

[12] , LKE (linode kubernetes engine)

[22] , LogSig

[23] , MoLFI (A search-based approach for accurate identification of log messages)

[24] , Spell (streaming parsing of system event logs)

[11] , and LogParse

[10] . Electronic devices can obtain the log parsing results shown in Table 3 using the log parsing methods of related technologies. As shown in Table 3:

[0128] Table 3

[0129] method F1-score Rand Index <![CDATA[Template (#templates a ) Quantity]]> FT-tree

[21] 0.4109 0.9423 188 IPLoM

[12] 0.4693 0.9552 125 LKE

[22] 0.0519 0.0266 3 LogSig

[23] 0.2815 0.8802 70 MoLFI

[24] 0.0914 0.4825 247 Spell

[11] 0.0953 0.5592 107 LogParse

[10] 0.6443 0.9808 535

[0130] The RandIndex score can be expressed as: RandIndex = (TP + TN) / (TP + TN + FP + FN). TP represents the sum of log pairs where x and y are manually classified into the same cluster and have the same prediction template, with x and y being two randomly selected logs; TN represents the sum of log pairs where x and y are manually classified into different clusters and have different prediction templates; FP represents the sum of log pairs where x and y are manually classified into different clusters but have the same prediction template; and FN represents the sum of log pairs where x and y are manually classified into the same cluster but have different prediction templates.

[0131] By using the log parsing method provided in the embodiment of the present application, the electronic device can obtain the log parsing results shown in Table 4. As shown in Table 4:

[0132] Table 4

[0133]

[0134]

[0135] It can be seen that compared with Table 3 and Table 4, the log parsing method provided by the embodiment of the present application can obtain more accurate log parsing results and a larger number of log parsing templates.

[0136] Moreover, in the process of the electronic device using the log parsing algorithm provided by the embodiment of the present application to perform log parsing, the classification model classifies the multiple first logs to obtain the classification results (such as Figure 6The grid-shaded areas shown in (a) and (b) are compared with the actual classification results of these first logs (such as Figure 6 Comparing the rectangular areas shown in (a) and (b) in the figure, it can be seen that the classification results output by the classification model are basically consistent with the actual classification results. Therefore, the classification results obtained by the classification model provided in the embodiment of the present application for classifying multi-source logs are relatively accurate and highly practical.

[0137] In addition, the running time of the electronic device using the log parsing algorithm provided by the embodiment of the present application is as follows: Figure 7 As shown, visible, in Figure 7 Compare the running time of the log parsing algorithm provided by the embodiment of the present application (such as Figure 7 The grid-shaded area shown in ) and the running time of the log parsing algorithm in the related art (such as Figure 7 ), the log parsing process of the embodiment of the present application takes less time and has higher log parsing efficiency.

[0138] like Figure 8 As shown, an embodiment of the present application further provides a log parsing device 800, which includes: an acquisition module 801 for acquiring multiple first logs, where the multiple first logs belong to different fields. An output module 802 for inputting the multiple first logs into a classification model, and outputting a domain identifier of the domain to which each first log in the multiple first logs belongs, wherein the classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs. A determination module 803 for determining a first log parsing template based on the second log parsing template corresponding to each domain identifier, wherein the first log parsing template is used to characterize the parsing results of multiple first logs in multiple fields, and the second log parsing template is used to characterize the parsing results of multiple first logs in a single field.

[0139] In this application, a classification model is used to classify multiple first logs to obtain the domain identifier of the domain to which each first log belongs, and the first log parsing templates of multiple first logs in multiple fields are determined based on the second log parsing templates corresponding to each domain identifier. It can be seen that this application converts the parsing problem of first logs in multiple fields into the parsing problem of first logs in multiple single domains, so that the parsing results obtained by parsing on each single domain are more accurate.

[0140] In one possible implementation, determination module 803 is configured to: determine a second log parsing template corresponding to the domain identifier based on the domain identifier and the first log corresponding to the domain identifier; and combine the second log parsing templates corresponding to the domain identifiers to obtain a first log parsing template. In this application, log parsing templates corresponding to each domain identifier are extracted based on the domain identifier and the set of logs corresponding to the domain identifier. In this way, extracting log parsing templates within a single domain will more fully explore the domain semantic features of the logs and obtain higher template quality.

[0141] In one possible implementation, the determination module 803 is used to: extract first key information and second key information of the first log corresponding to the domain identifier, the first key information is used to characterize the attribute information of the domain to which the domain identifier belongs, and the second key information is used to characterize the parameter value corresponding to the attribute information; extract key features from the first key information and the second key information, the key features being the features with the highest frequency of occurrence; and determine the second log parsing template corresponding to the domain identifier based on the key features.

[0142] In one possible implementation, the log parsing device 800 also includes: a training module 804; an acquisition module 801 for acquiring a reference number of second logs and domain identifiers corresponding to the second logs. The training module 804 is used to train a classification model using the reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data. In the present application, the classification model can be trained on logs from multiple different source fields. The classification model can transfer domain-specific semantic knowledge in most multi-source log sets without supervision. It greatly reduces the workload of manual labeling through unsupervised transfer learning, further reducing the implementation cost of log processing.

[0143] In one possible implementation, acquisition module 801 is configured to obtain a reference number of third logs; if the third logs include invalid information, delete the invalid information from the third logs to obtain the second logs, where the invalid information is information in the third logs that does not contribute to the generation of the first log parsing template. In this embodiment of the present application, by deleting specific symbols contained in the logs, the generalization capability of the classification model can be effectively improved, thereby enhancing the versatility of the classification model.

[0144] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0145] In one possible implementation, acquisition module 801 is configured to: obtain a reference number of third logs; and, if the third logs include irregular characters, correct the irregular characters in the third logs using a regular expression to obtain a second log. In this embodiment of the present application, by correcting the irregular characters in the third logs using a regular expression, the information in the second logs used to train the classification model is made more concise, effectively eliminating interfering information, and thereby making the classification model trained using the second logs more accurate.

[0146] In one possible implementation, acquisition module 801 is configured to: obtain a reference number of third logs; if the third logs include phrases containing special characters, split the phrases in the third logs and remove the special characters from the phrases to obtain the second logs. In this embodiment of the present application, by splitting the phrases and removing the special characters from the phrases, the processed phrases are more readable, facilitating the provision of accurate training data for subsequent analysis model training, thereby improving the accuracy of model training.

[0147] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0148] like Figure 9 As shown, an embodiment of the present application also provides a training device 900 for a classification model, which includes: an acquisition module 901, used to obtain a reference number of second logs and domain identifiers corresponding to the second logs; a training module 902, used to use the reference number of second logs as input data, and use the domain identifier corresponding to each second log in the reference number of second logs as output data to train a classification model, wherein the classification model is used to distinguish the domain identifier of the domain to which each first log in multiple first logs belongs, and the multiple first logs belong to different fields.

[0149] In an embodiment of the present application, a classification model is trained by using a reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data. The resulting classification model can accurately identify the domain to which the logs belong and remains robust to domain classification errors. Based on this classification model, the foundation is laid for the subsequent generation of log parsing templates for multi-domain logs, significantly improving the accuracy and efficiency of subsequent log parsing.

[0150] In one possible implementation, acquisition module 901 is configured to: obtain a reference number of third logs; if the third logs include invalid information, delete the invalid information from the third logs to obtain the second logs, where the invalid information is information in the third logs that does not contribute to the generation of a first log parsing template, where the first log parsing template is used to represent the parsing results of multiple first logs in multiple fields. In this embodiment of the present application, by deleting specific symbols contained in the logs, the generalization capability of the classification model can be effectively improved, thereby enhancing the versatility of the classification model.

[0151] In a possible implementation, the invalid information includes at least one of the following: time information, identification information, a specific symbol, or log header information.

[0152] In one possible implementation, acquisition module 901 is configured to: obtain a reference number of third logs; and, if the third logs include irregular characters, correct the irregular characters in the third logs using a regular expression to obtain a second log. In this embodiment of the present application, by correcting the irregular characters in the third logs using a regular expression, the information in the second logs used to train the classification model is made more concise, effectively eliminating interfering information, and thereby making the classification model trained using the second logs more accurate.

[0153] In one possible implementation, acquisition module 901 is configured to: obtain a reference number of third logs; if the third logs include phrases containing special characters, split the phrases in the third logs and remove the special characters from the phrases to obtain the second logs. In this embodiment of the present application, by splitting the phrases and removing the special characters from the phrases, the processed phrases are more readable, facilitating the provision of accurate training data for subsequent analysis model training, thereby improving the accuracy of model training.

[0154] It should be understood that the above Figure 9 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0155] See also Figure 10 , Figure 10 A schematic structural diagram of an electronic device 1000 provided by an exemplary embodiment of the present application is shown. Figure 10 The electronic device 1000 shown is used to perform the above Figure 5 The operations involved in the log parsing method shown in FIG. 1 . The electronic device 1000 can be implemented by a general bus architecture.

[0156] like Figure 10 As shown, the electronic device 1000 includes at least one processor 1001 , a memory 1003 and at least one communication interface 1004 .

[0157] The processor 1001 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing units (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 1001 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can implement or execute the various logic blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0158] Optionally, the electronic device 1000 further includes a bus. The bus is used to transmit information between the components of the electronic device 1000. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0159] The memory 1003 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, or a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1003 is, for example, independent and connected to the processor 1001 via a bus. The memory 1003 can also be integrated with the processor 1001.

[0160] The communication interface 1004 uses any transceiver-like device for communicating with other devices or communication networks. The communication network can be Ethernet, a radio access network (RAN), or a wireless local area network (WLAN). The communication interface 1004 can include a wired communication interface and a wireless communication interface. Specifically, the communication interface 1004 can be an Ethernet interface, a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, an asynchronous transfer mode (ATM) interface, a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In the embodiment of the present application, the communication interface 1004 can be used for the electronic device 1000 to communicate with other devices.

[0161] In a specific implementation, as an embodiment, the processor 1001 may include one or more CPUs, such as Figure 10 0 and CPU1 are shown in FIG. Each of these processors can be a single-CPU processor or a multi-CPU processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] In a specific implementation, as an embodiment, the electronic device 1000 may include multiple processors, such as Figure 10 1 and 1005. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0163] In a specific implementation, as an embodiment, the electronic device 1000 may further include an output device and an input device. The output device communicates with the processor 1001 and can display information in a variety of ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 1001 and can receive user input in a variety of ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0164] In some embodiments, the memory 1003 is used to store the program code 1010 for executing the solution of the present application, and the processor 1001 can execute the program code 1010 stored in the memory 1003. That is, the electronic device 1000 can implement the log parsing method provided by the method embodiment through the processor 1001 and the program code 1010 in the memory 1003. The program code 1010 may include one or more software modules. Optionally, the processor 1001 itself may also store the program code or instructions for executing the solution of the present application.

[0165] In a specific embodiment, the electronic device 1000 of the embodiment of the present application may correspond to the computing device in the above-mentioned various method embodiments.

[0166] in, Figure 5 Each step of the log parsing method shown is completed by the hardware integrated logic circuit or software instructions in the processor of the electronic device 1000. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0167] The present application also provides an electronic device including a processor configured to load and execute at least one instruction to enable the electronic device to implement the method provided in the present application. Optionally, the device also includes a memory coupled to the processor and configured to store the at least one instruction.

[0168] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0169] The embodiments of the present application further provide a computer program (product), which, when executed by a computer, can enable a processor or computer to execute the corresponding steps and / or processes in the above method embodiments.

[0170] An embodiment of the present application further provides a chip, which includes a processor for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes any of the above methods.

[0171] An embodiment of the present application also provides another chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute any of the methods described above.

[0172] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0173] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the setting results involved in this application are all obtained with full authorization.

[0174] Those skilled in the art will appreciate that the various method steps and modules described in conjunction with the embodiments disclosed herein can be implemented in software, hardware, firmware, or any combination thereof. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0176] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. As an example, the method of the embodiment of the present application can be described in the context of a machine executable instruction, and the machine executable instruction is such as included in the program module executed in the device on the real or virtual processor of the target. Generally speaking, a program module includes a routine, a program, a library, an object, a class, a component, a data structure, etc., which performs a specific task or realizes a specific abstract data structure. In various embodiments, the function of the program module can be merged or split between the described program modules. The machine executable instruction for the program module can be executed in a local or distributed device. In a distributed device, the program module can be located in both a local and a remote storage medium.

[0177] The computer program code for implementing the method of the embodiment of the present application can be written in one or more programming languages. These computer program codes can be provided to the processor of the determination device of a general-purpose computer, a special-purpose computer or other programmable proxy node, so that when the program code is executed by the determination device of the computer or other programmable proxy node, the function / operation specified in the flow chart and / or block diagram is implemented. The program code can be executed entirely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0178] In the context of the embodiments of the present application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like.

[0179] Examples of signals may include electrical, optical, radio, acoustic or other forms of propagated signals, such as carrier waves, infrared signals, etc.

[0180] A machine-readable medium may be any tangible medium that contains or stores a program for or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of machine-readable storage media include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the above-described systems, devices, and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or can be electrical, mechanical or other forms of connection.

[0183] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0184] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0185] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0186] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limit on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first log can be referred to as the second log, and similarly, the second log can be referred to as the first log. The first log and the second log can both be logs, and in some cases, can be separate and different logs.

[0187] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0188] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, "plurality of second logs" means two or more second logs. The terms "system" and "network" are often used interchangeably herein.

[0189] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0190] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The term "and / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0191] It will also be understood that the term “comprise” (also known as “includes,” “including,” “comprises,” and / or “comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0192] It should also be understood that the terms “if” and “if” may be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting.” Similarly, the phrases “if it is determined that ” or “if [stated condition or event] is detected” may be interpreted to mean “upon determining ” or “in response to determining ” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.

[0193] It should be understood that determining B based on A does not mean determining B based solely on A. B can also be determined based on A and / or other information.

[0194] It should also be understood that references throughout this specification to "one embodiment," "an embodiment," or "one possible implementation" mean that specific features, structures, or characteristics associated with that embodiment or implementation are included in at least one embodiment of the present application. Therefore, the appearance of "in one embodiment," "in an embodiment," or "one possible implementation" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Claims

1. A log parsing method, characterized in that: The method comprises: Acquire multiple first logs, where the multiple first logs belong to different fields; inputting the plurality of first logs into a classification model, and outputting a domain identifier of a domain to which each of the plurality of first logs belongs, wherein the classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs; A first log parsing template is determined based on the second log parsing template corresponding to each domain identifier. The first log parsing template is used to represent the parsing results of multiple first logs in multiple fields, and the second log parsing template is used to represent the parsing results of multiple first logs in a single field.

2. The method according to claim 1, characterized in that Before determining the first log parsing template, the method further includes: Determining a second log parsing template corresponding to the domain identifier according to the domain identifier and the first log corresponding to the domain identifier; Determining the first log parsing template includes: The second log parsing templates corresponding to the domain identifiers are combined to obtain the first log parsing template.

3. The method according to claim 2, characterized in that The determining of the second log parsing template corresponding to the domain identifier includes: Extracting first key information and second key information of the first log corresponding to the domain identifier, where the first key information is used to represent attribute information of the domain to which the domain identifier belongs, and the second key information is used to represent a parameter value corresponding to the attribute information; Extracting key features from the first key information and the second key information, wherein the key features are features with the highest frequency of occurrence; The second log parsing template corresponding to the domain identifier is determined according to the key feature.

4. The method according to any one of claims 1 to 3, characterized in that Before inputting the plurality of first logs into the classification model and outputting the domain identifier of the domain to which each of the plurality of first logs belongs, the method further includes: Obtain a reference number of second logs and domain identifiers corresponding to the second logs; The classification model is trained by using the reference number of second logs as input data and using the domain identifier corresponding to each second log in the reference number of second logs as output data.

5. The method according to claim 4, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case that the third log includes invalid information, the invalid information in the third log is deleted to obtain the second log, where the invalid information is information in the third log that does not contribute to the generation of the first log parsing template.

6. The method according to claim 5, characterized in that The invalid information includes at least one of the following: time information, identification information, specific symbols or log header information.

7. The method according to claim 4, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case that the third log includes irregular characters, regular expressions are used to correct the irregular characters in the third log to obtain the second log.

8. The method according to claim 4, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case that the third log includes a phrase containing special characters, the phrases in the third log are split, and the special characters in the phrases are removed to obtain the second log.

9. A classification model training method, characterized in that: The method comprises: Obtain a reference number of second logs and domain identifiers corresponding to the second logs; The reference number of second logs is used as input data, and the domain identifier corresponding to each second log in the reference number of second logs is used as output data to train a classification model, wherein the classification model is used to distinguish the domain identifier of the domain to which each first log in multiple first logs belongs, and the multiple first logs belong to different fields.

10. The method according to claim 9, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case where the third log includes invalid information, the invalid information in the third log is deleted to obtain the second log, where the invalid information is information in the third log that does not contribute to the generation of the first log parsing template, and the first log parsing template is used to represent the parsing results of multiple first logs in multiple fields.

11. The method according to claim 10, characterized in that The invalid information includes at least one of the following: time information, identification information, specific symbols or log header information.

12. The method according to claim 9, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case that the third log includes irregular characters, regular expressions are used to correct the irregular characters in the third log to obtain the second log.

13. The method according to claim 9, characterized in that The obtaining of a reference number of second logs includes: obtaining a third log of the reference quantity; In the case that the third log includes a phrase containing special characters, the phrases in the third log are split, and the special characters in the phrases are removed to obtain the second log.

14. A log analysis device, characterized in that: The device comprises: An acquisition module, configured to acquire a plurality of first logs, wherein the plurality of first logs belong to different fields; an output module, configured to input the plurality of first logs into a classification model and output a domain identifier of a domain to which each of the plurality of first logs belongs, wherein the classification model is trained based on a reference number of second logs and the domain identifiers corresponding to the second logs; A determination module is used to determine a first log parsing template based on a second log parsing template corresponding to each of the domain identifiers, wherein the first log parsing template is used to represent the parsing results of multiple first logs in multiple fields, and the second log parsing template is used to represent the parsing results of multiple first logs in a single field.

15. A training device for a classification model, characterized in that: The device comprises: an acquisition module, configured to acquire a reference number of second logs and domain identifiers corresponding to the second logs; A training module is used to use the reference number of second logs as input data and the domain identifier corresponding to each second log in the reference number of second logs as output data to train a classification model, wherein the classification model is used to distinguish the domain identifier of the domain to which each first log in a plurality of first logs belongs, and the plurality of first logs belong to different fields.

16. An electronic device, characterized in that: The electronic device includes a memory and a processor; the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to enable the electronic device to implement any one of the methods described in claims 1-8, or any one of the methods described in claims 9-13.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 8, or to implement the method according to any one of claims 9 to 13.

18. A computer program product, characterized in that The computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor to enable a computer to implement the method according to any one of claims 1 to 8, or to implement the method according to any one of claims 9 to 13.

Images