Method, device and electronic device for determining parsing rules

Abstract

The present application relates to the field of computer technology, and specifically to a method, device and electronic device for determining parsing rules, which are used to solve the problems of low efficiency and inaccuracy caused by manually determining parsing rules in related technologies. In this method, the number of target variables of the log to be accessed is first determined, and the number of target variables is used as the target length of the log to be accessed. Then, based on the target length and the access type of the log to be accessed, N types of parsing rules corresponding to the log to be accessed are determined. Then, these N types of parsing rules are used to parse the log to be accessed respectively to obtain N parsing results. Finally, among these N parsing results, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule corresponding to the log to be accessed. Based on the above method, the parsing rules can be automatically determined, thereby improving the accuracy and efficiency of the parsing rules finally determined.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and electronic device for determining parsing rules. Background Art

[0002] When logs are transmitted to log access components or third-party systems, corresponding parsing rules need to be used to parse the logs. Currently, log parsing is generally done manually, that is, the corresponding parsing rules are determined manually based on personal experience, and then the association between the logs and the corresponding parsing rules is established.

[0003] However, in actual application scenarios, determining the parsing rules corresponding to the logs to be accessed requires high levels of human professional ability, which will increase the corresponding learning and labor costs. In addition, during the process of manual participation, there is also the problem of inaccurate parsing rules due to differences in personal experience. Summary of the Invention

[0004] The present application provides a method, device and electronic device for determining parsing rules, which are used to reduce manual dependence and improve the efficiency and accuracy of determining parsing rules.

[0005] In a first aspect, the present application provides a method for determining a parsing rule, the method comprising:

[0006] Determining the number of target variables of the log to be accessed, and using the number of target variables as the target length of the log to be accessed; wherein the target variables represent the field types in the log to be accessed;

[0007] Determining, according to the target length and the access type of the log to be accessed, N parsing rules corresponding to the log to be accessed; wherein N is a positive integer greater than or equal to 1;

[0008] Using the N parsing rules, respectively parse the log to be accessed to obtain N parsing results; wherein one parsing result includes multiple parsing fields;

[0009] Among the N parsing results, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule corresponding to the log to be accessed.

[0010] As a possible implementation method, determining N parsing rules corresponding to the log to be accessed based on the target length and the access type of the log to be accessed includes: determining a first log set corresponding to the log to be accessed based on the target length and the access type of the log to be accessed; wherein the first log set contains at least one sample log; calculating the similarity between the target variable and the reference variable of each sample log in the first log set; selecting all sample logs corresponding to similarities greater than a preset threshold as target sample logs; and using the N parsing rules corresponding to all target sample logs as the N parsing rules corresponding to the log to be accessed.

[0011] As a possible implementation method, determining the first log set corresponding to the log to be accessed based on the target length and the access type of the log to be accessed includes: determining a second log set based on the target length; wherein the reference length of the sample logs in the second log set is the same as the target length; and in the second log set, taking the sample logs with the same access type as the log to be accessed as the first log set.

[0012] As a possible implementation, the calculation of the similarity between the target variable and the reference variables of each sample log in the first log set includes: determining the target variable sequence of the log to be accessed based on the arrangement order of the target variable in the log to be accessed; in the first log set, determining the reference variable sequence of each sample log based on the arrangement order of the reference variables of each sample log in the each sample log; and using the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log.

[0013] As a possible implementation manner, using the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log includes: respectively calculating the longest common subsequence between the target variable sequence and the reference variable sequence of each sample log; and using the length of the respectively calculated longest common subsequence as the similarity between the target variable and the reference variable of each sample log.

[0014] In a second aspect, the present application provides a device for determining a parsing rule, the device comprising:

[0015] a target length determination module, determining the number of target variables of the log to be accessed, and using the number of target variables as the target length of the log to be accessed; wherein the target variables represent the field types in the log to be accessed;

[0016] A first parsing rule determination module is configured to determine N parsing rules corresponding to the log to be accessed based on the target length and the access type of the log to be accessed, wherein N is a positive integer greater than or equal to 1;

[0017] Obtaining a parsing result module, using the N parsing rules to parse the log to be accessed respectively, to obtain N parsing results; wherein one parsing result includes multiple parsing fields;

[0018] The second parsing rule determination module uses, among the N parsing results, a parsing rule corresponding to a parsing result containing the most parsing fields as a target parsing rule corresponding to the log to be accessed.

[0019] As a possible implementation method, the N parsing rules corresponding to the log to be accessed are determined based on the target length and the access type of the log to be accessed. The first parsing rule determination module is specifically used to: determine the first log set corresponding to the log to be accessed based on the target length and the access type of the log to be accessed; wherein the first log set contains at least one sample log; calculate the similarity between the target variable and the reference variable of each sample log in the first log set; select all sample logs corresponding to the similarity greater than a preset threshold as target sample logs; and use the N parsing rules corresponding to all target sample logs as the N parsing rules corresponding to the log to be accessed.

[0020] As a possible implementation method, the first log set corresponding to the log to be accessed is determined based on the target length and the access type of the log to be accessed, and the first determination parsing rule module is specifically used to: determine the second log set based on the target length; wherein the reference length of the sample log in the second log set is the same as the target length; in the second log set, the sample log with the same access type as the log to be accessed is used as the first log set.

[0021] As a possible implementation method, the similarity between the target variable and the reference variables of each sample log in the first log set is calculated, and the first determination parsing rule module is specifically used to: determine the target variable sequence of the log to be accessed based on the arrangement order of the target variable in the log to be accessed; determine the reference variable sequence of each sample log in the first log set based on the arrangement order of the reference variables of each sample log in the each sample log; and use the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log.

[0022] As a possible implementation method, the similarity between the target variable sequence and the reference variable sequence of each sample log is used as the similarity between the target variable and the reference variable of each sample log. The first determination parsing rule module is specifically used to: respectively calculate the longest common subsequence between the target variable sequence and the reference variable sequence of each sample log; and use the length of the longest common subsequence calculated respectively as the similarity between the target variable and the reference variable of each sample log.

[0023] In a third aspect, the present application provides an electronic device, comprising:

[0024] Memory for storing computer programs;

[0025] The processor is configured to implement the above-mentioned method steps for determining a parsing rule when executing the computer program stored in the memory.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned method steps for determining a parsing rule.

[0027] In an embodiment of the present application, the number of target variables of the log to be accessed, that is, the target length of the log to be accessed, is first determined, and then based on the target length of the log to be accessed and the access type of the log to be accessed, N kinds of parsing rules corresponding to the log to be accessed are determined, and finally the one with the best parsing effect is selected from these N kinds of parsing rules as the target parsing rule. On the one hand, by using the target length and access type of the log to be accessed, N parsing rules corresponding to the log to be accessed can be determined from a large number of parsing rules, solving the problem that the related technology requires manual intervention to match the log to be accessed with a large number of parsing rules, thereby realizing the automatic determination of the target parsing rules and improving the efficiency of the determination process. On the other hand, N kinds of parsing rules are used to parse the log to be accessed respectively, and N kinds of parsing results are obtained. Then, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule. In this way, the log to be accessed can be intelligently and automatically matched to the target parsing rule with the highest resolution, thereby effectively improving the resolution and accuracy of the determined target parsing rule.

[0028] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a possible implementation environment provided for this application;

[0030] Figure 2 A flowchart of a method for determining parsing rules provided in this application;

[0031] Figure 3 A schematic diagram of a preset threshold setting mechanism provided in this application;

[0032] Figure 4 A schematic diagram of a device for determining parsing rules provided in this application;

[0033] Figure 5 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0035] In the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] First, some terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0037] Log: A record of information designed to meet system requirements, specifically used to record system operations or service content, such as IP (Internet Protocol) addresses, port protocols, time, usernames, etc. For example, in the embodiments of this application, logs can be understood as operational or service records of third-party security devices or network devices.

[0038] Log parsing: Its purpose is to convert unstructured raw logs into structured data. For example, in the embodiment of the present application, log parsing can be understood as performing normalization operations such as parsing, enrichment, and conversion on the raw logs according to the log format.

[0039] Log access: refers to reporting logs and processing them, and saving the final processing results. For example, in the embodiment of the present application, log access can be understood as the whole process of normalizing the logs through the corresponding parsing rules and then persisting the logs.

[0040] Parsing rules: Parsing rules are used to identify text patterns of specific concepts, that is, key information in the log can be identified through parsing rules. For example, in the embodiment of the present application, parsing rules can be understood as a set of parsing operations that need to be performed during the entire log parsing process.

[0041] It should be noted that this solution can be applied to determine target parsing rules for logs to be accessed in a variety of application scenarios, such as security systems, security platforms, information security, and log access. This solution can also be applied to tasks that require improving the accuracy of target parsing rules for logs to be accessed.

[0042] This solution can be implemented on computing devices such as computing terminals and remote servers. By deploying it on relevant computing devices and based on the number of target variables and access type of the logs to be accessed, the accuracy of the target parsing rules corresponding to the logs to be accessed can be improved. This is merely an illustrative example of the application of this solution and does not specifically limit it.

[0043] The following is a brief introduction to the design concept of the method for determining parsing rules provided in the embodiment of the present application.

[0044] In the log access scenario, related technologies use log access components or third-party systems. This requires the user to have a full understanding of the original log text format and the parsing rules of the log access components or third-party systems. In other words, the user must manually participate in mapping and associating the log text corresponding to the parsing rules.

[0045] It is easy to understand that the log access component or third-party system still relies on human participation. Specifically, when accessing logs with parsing, the user needs to have some understanding or foreknowledge of the original format of the log, and also have a full understanding of the parsing rules of the log access component or third-party system. Only then can the user, based on the individual experience of the user, map the associated parsing rules with the original log of the corresponding type to obtain the final target parsing rules. For example, based on the individual experience of the user, the source identifier of the log to be accessed (such as: manufacturer, device type, version, etc.) is matched to obtain the target parsing rules corresponding to the log to be accessed. However, in actual applications, determining the parsing rules corresponding to the log to be accessed requires high professional ability of humans, which will increase the corresponding learning cost and labor cost. In addition, in the process of manual participation, there is also the problem of inaccurate parsing rules due to differences in personal experience. Therefore, the relevant technology cannot better solve the problem of requiring human participation in the association of log originals and parsing rules, and there is a problem of high knowledge requirements for operators.

[0046] It can be seen that in the related art, there is a problem of inaccurate target parsing rules for determining the logs to be accessed.

[0047] In order to improve the accuracy of the determined target parsing rules, this application proposes a method for determining parsing rules. In this method, the number of target variables of the log to be accessed is first determined, and the number of target variables is used as the target length of the log to be accessed. Then, based on the target length and the access type of the log to be accessed, N parsing rules corresponding to the log to be accessed are determined. Then, these N parsing rules are used to parse the log to be accessed respectively to obtain N parsing results. Finally, among these N parsing results, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule corresponding to the log to be accessed.

[0048] The target variable represents the field type in the log to be accessed; N is a positive integer greater than or equal to 1; and a parsing result includes multiple parsing fields.

[0049] In an embodiment of the present application, the number of target variables of the log to be accessed, that is, the target length of the log to be accessed, is first determined, and then based on the target length of the log to be accessed and the access type of the log to be accessed, N kinds of parsing rules corresponding to the log to be accessed are determined, and finally the one with the best parsing effect is selected from these N kinds of parsing rules as the target parsing rule. On the one hand, by using the target length and access type of the log to be accessed, N parsing rules corresponding to the log to be accessed can be determined from a large number of parsing rules, solving the problem that the related technology requires manual intervention to match the log to be accessed with a large number of parsing rules, thereby realizing the automatic determination of the target parsing rules and improving the efficiency of the determination process. On the other hand, N kinds of parsing rules are used to parse the log to be accessed respectively, and N kinds of parsing results are obtained. Then, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule. In this way, the log to be accessed can be intelligently and automatically matched to the target parsing rule with the highest resolution, thereby effectively improving the resolution and accuracy of the determined target parsing rule.

[0050] The method for determining parsing rules provided in the embodiment of the present application can be applied to Figure 1 In the implementation environment shown, the implementation environment may include at least an access node, an operation node, a management node, a computing node, and a storage node.

[0051] The access node is used to obtain the log to be accessed.

[0052] The operation node is used to interact with the user so that the user can deploy, configure, and manage the target parsing rule tasks.

[0053] The management node is used to obtain the logs to be accessed from the access node, for example, see Figure 1The access node uploads the pending access log to the cloud, and the management node obtains the pending access log from the cloud. The management node also manages the compute nodes and storage nodes in conjunction with the target parsing rule tasks. During the management process, the management node forwards the pending access log to the compute nodes.

[0054] The computing node is used to complete the computing tasks involved in determining the target parsing rule task based on the received logs to be accessed, thereby achieving computing acceleration.

[0055] The storage node is used to store the logs to be accessed, various preset parsing rules and various mapping relationships generated by determining the target parsing rules (for example: N types of parsing rules corresponding to the logs to be accessed, target parsing rules corresponding to the logs to be accessed, etc.) according to the management of the management node for tracing.

[0056] It should be noted that the access node, management node, computing node, and storage node described above are different devices, or any two or three of the management node, computing node, and storage node can also be integrated into the same device. The above-mentioned operation node is not required and is not specifically limited in this solution.

[0057] The following is a detailed introduction to the method for determining parsing rules provided in the embodiment of the present application. Figure 2 The method includes steps 201 to 204, which are specifically as follows.

[0058] Step 201: Determine the number of target variables of the log to be accessed, and use the number of target variables as the target length of the log to be accessed;

[0059] The target variable represents the field type in the log to be accessed. To more clearly describe the meaning of the target variable, the following is a necessary explanation of the log characteristics of the log to be accessed.

[0060] Logs can include text format description statements and system variables. The format description statements of the same log are usually fixed, while the system variables change with the changes in log parameters. For example, a log feature can be as follows:

[0061] “ <46> Nov 08 01:05:33linux messages[linux.host]:dev_ip=****;os=SUSELinux Enterprise Server 11SP4;name=messages;object=lsyslog;msg=Nov 801:05:32linux--MARK–”;

[0062] Among them, "46", "Nov 08 01:05:33", "linux.host", "****", "11SP4" and "Nov8 01:05:32" are used to identify the content, that is, system variables, and the rest are used to identify the format, that is, text description statements.

[0063] As an optional implementation method, before determining the number of target variables of the log to be accessed, normalization and feature vector extraction can also be performed on the log to be accessed. The key to this process is to retain the format description statement of the log to be accessed and ignore the system variables. From the exemplary sample of the aforementioned log feature, it can be seen that the system variables mainly include: IP, port protocol and other data; digital information and time variables. In an embodiment of the present application, a corresponding feature vector library can be selected from a preset feature vector extraction library to perform normalization and feature vector extraction operations on the log to be accessed. Common feature vector extraction libraries can be exemplified as follows:

[0064] 1)msg=re.sub('(\d+\.)+\d+','IP',msg)

[0065] Used to filter IP format data;

[0066] 2)msg=re.sub('Mar|Apr|Dec|Jan|Feb|Nov|Oct|May|Jun|Jul|Aug|Sep','M',msg)

[0067] Used to filter the time parameter month;

[0068] 3)msg=re.sub('\\d{d}:\\d{2}:\\d{2}','H',msg)

[0069] Used to filter time parameters hours, minutes, and seconds;

[0070] 4)msg=re.sub('\\d{2,4}[\ / -]\\d{2}[\ / -]\\d{2,4}','Y',msg)

[0071] Used to filter the time parameters year, month, and day 2020-12-11 and 2020 / 12 / 11;

[0072] 5)msg=re.sub('(=[1-9][0-9]*)','=p',msg)

[0073] Used to filter continuous pure numbers;

[0074] 6)msg=re.sub('\\b(0[xX])?[A-Fa-f0-9]+\\b','X',msg)

[0075] Used to filter hexadecimal memory addresses.

[0076] Of course, the above six points are only an exemplary feasible method. The embodiment of this application does not specifically limit the feature vector extraction library. The feature vector extraction library can also be supported or supplemented according to actual application requirements to reserve scalability to support more types of feature vector data formats.

[0077] Furthermore, after normalization and feature vector extraction are performed on the log to be accessed, the log to be accessed can be divided into a constant portion and a variable portion. For easy understanding, the processed text format description statements of the log to be accessed are collectively referred to as the constant portion, while the processed system variables of the log to be accessed are collectively referred to as the variable portion. This allows the target number of variables in the log to be accessed to be determined based on the variable portion, and this number of variables serves as the target length of the log to be accessed.

[0078] Typically, the target variable is the result of normalizing and extracting the eigenvectors of the system variables to be accessed. The number of target variables is the number of variables in the result. However, target variables can also be filtered or excluded based on actual application needs, which is not explained here.

[0079] In view of this, in order to minimize the matching errors caused by different log sample formats, step 201 of the embodiment of the present application mainly focuses on the target length of the log to be accessed, that is, the number of target variables, which helps to improve the accuracy of the final target parsing rules.

[0080] Step 202: Determine N parsing rules corresponding to the log to be accessed based on the target length and the access type of the log to be accessed;

[0081] Wherein, N is a positive integer greater than or equal to 1.

[0082] In an embodiment of the present application, a first log set corresponding to the log to be accessed, i.e., a set containing at least one sample log, can be determined based on the target length of the log to be accessed and the access type of the log to be accessed. Then, the similarity between the target variable and the reference variable of each sample log in the first log set is calculated. All sample logs corresponding to similarities greater than a preset threshold are selected as target sample logs. Finally, the N parsing rules corresponding to all target sample logs are used as the N parsing rules corresponding to the log to be accessed.

[0083] The above process of determining the first log set corresponding to the log to be accessed can be understood as selecting the first log set corresponding to the target length of the log to be accessed from several log sets according to the access type of the log to be accessed. That is to say, among the several log sets, each log set not only corresponds to an access type, but also corresponds to a target length. Then, based on the target length and access type of the log to be accessed, the corresponding first log set is determined. In this way, a medium range can be defined from a larger range, that is, from several log set levels to the log set level. In other words, the matching relationship between each log set and the target length can be understood as a hierarchical clustering idea of ​​length matching. Based on this, deep integration is performed according to the access type to determine the first log set corresponding to the log to be accessed. This can improve the matching accuracy for the subsequent further matching of parsing rules and effectively improve the matching performance.

[0084] In more detail, as a possible implementation method, the first log set corresponding to the log to be accessed is determined based on the target length and the access type of the log to be accessed. The specific steps may include: based on the target length, determining the second log set, that is, the reference length of the sample log in the second log set is the same as the target length; then, in the second log set, the sample log with the same access type as the log to be accessed is used as the first log set.

[0085] After determining the middle range of the first log set, the range needs to be further narrowed. In an embodiment of the present application, the target sample log is determined by calculating the similarity between the target variable and the reference variable of each sample log in the first log set, and then the N parsing rules corresponding to the target sample log are used as the N parsing rules corresponding to the log to be accessed.

[0086] In detail, as a possible implementation method, the similarity between the target variable and the reference variable of each sample log in the first log set is calculated. The specific steps may include: determining the target variable sequence of the log to be accessed based on the arrangement order of the target variable in the log to be accessed, and then determining the reference variable sequence of each sample log in the first log set based on the arrangement order of the reference variables of each sample log in each sample log, and then using the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log.

[0087] In more detail, as a possible implementation method, the similarity between the target variable sequence and the reference variable sequence of each sample log is used as the similarity between the target variable and the reference variable of each sample log. The specific steps may include: respectively calculating the longest common subsequence between the target variable sequence and the reference variable sequence of each sample log, and then using the length of the respectively calculated longest common subsequence as the similarity between the target variable and the reference variable of each sample log.

[0088] Furthermore, based on the similarity between the target variable and the reference variable of each sample log, all sample logs corresponding to similarities greater than a preset threshold are selected as target sample logs, and the parsing rules corresponding to the target sample logs are used as the parsing rules corresponding to the logs to be accessed.

[0089] In view of this, we first preliminarily determine the first set of logs for similar processing through length matching, and then obtain the similarity between the target variable and the reference variable of each sample log by calculating the longest common subsequence. Then, we compare the similarity with the preset threshold, and obtain the parsing rule corresponding to the optimal target sample log based on the comparison result. Here, we automatically establish the association relationship between these parsing rules and the log to be accessed, so as to achieve the effect of intelligent association of sample logs and log parsing rule mapping, and solve the problem of low efficiency caused by manual participation in related technologies.

[0090] Furthermore, the calculation of similarity is influenced by many factors. For example, different device logs may have varying matching similarities due to differences in log format, content, and length. Furthermore, the matching similarities of the same device log may also vary due to differences in log type and length. Therefore, the setting of the preset threshold for similarity directly affects the accuracy of the resulting parsing rules. To improve this accuracy, we propose a mechanism for setting the preset threshold for training. The following supplementary explanation is provided regarding this mechanism.

[0091] As a supplementary explanation, the preset threshold can support one of the following two methods and their combination. Method one, one preset threshold for one parsing rule, that is, one parsing rule can correspond to multiple sample logs, that is, the preset threshold of the parsing rule can be obtained by calculating the similarity based on multiple sample logs. Method two, a preset threshold is set as a whole, that is, each parsing rule corresponds to only one sample log, then a common overall threshold can be trained for it as the preset threshold. In other words, corresponding to a parsing rule, the same parsing rule can correspond to multiple sample logs, that is, a threshold can be specified for each sample, or a threshold can be uniformly specified for all sample logs corresponding to this parsing rule.

[0092] For example, Figure 3The figure below shows a schematic diagram combining Method 1 and Method 2. For example, if parsing rule 1 corresponds to sample logs 1, 2, and 3, then Threshold 1 for parsing rule 1 can be obtained for sample logs 1, 2, and 3. Similarly, Threshold 2 for parsing rule 2 and Threshold 3 for parsing rule 3 can be obtained, and their details are omitted. Furthermore, a common overall threshold can be trained based on these methods.

[0093] It should be noted that the source of the preset threshold can be obtained through training. The training threshold is related to the log component or the system's rule base, and is calculated by performing multi-dimensional evaluation of all rules in the rule base (for example: the number of fields that can be parsed by each parsing rule, and calculating the average value).

[0094] Optionally, in some embodiments, if the first log set corresponding to the log to be accessed cannot be determined, that is, no log set has a target length that matches the target length of the log to be accessed, then a missing special partition can be assigned to the log to be accessed, and a corresponding prompt can be provided to the user as a reminder. Furthermore, the missing special partition can also correspond to a corresponding parsing rule. In this case, the parsing rule corresponding to the missing special partition can be directly used as the parsing rule for the log to be accessed.

[0095] Step 203: Using N parsing rules, parse the access log to obtain N parsing results.

[0096] Among them, a parsing result includes multiple parsing fields. If the parsing result contains more parsing fields, it proves that the parsing accuracy of the parsing rule corresponding to the parsing result is higher.

[0097] Step 204: Among the N parsing results, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule corresponding to the log to be accessed.

[0098] Furthermore, the logs to be accessed are automatically associated with the target parsing rules to achieve the technical effect of intelligently associating the target parsing rules.

[0099] Based on the same inventive concept, the present application also provides a device for determining parsing rules, for Figure 4 , the device comprises:

[0100] The target length determination module 401 determines the number of target variables of the log to be accessed, and uses the number of target variables as the target length of the log to be accessed; wherein the target variables represent the field types in the log to be accessed;

[0101] A first parsing rule determination module 402 determines N parsing rules corresponding to the log to be accessed according to the target length and the access type of the log to be accessed, wherein N is a positive integer greater than or equal to 1;

[0102] Obtaining parsing result module 403, using the N parsing rules to parse the log to be accessed respectively, to obtain N parsing results; wherein one parsing result includes multiple parsing fields;

[0103] The second parsing rule determination module 404 uses, among the N parsing results, a parsing rule corresponding to a parsing result containing the most parsing fields as a target parsing rule corresponding to the log to be accessed.

[0104] As a possible implementation method, the N parsing rules corresponding to the log to be accessed are determined based on the target length and the access type of the log to be accessed. The first parsing rule determination module 402 is specifically used to: determine the first log set corresponding to the log to be accessed based on the target length and the access type of the log to be accessed; wherein the first log set contains at least one sample log; calculate the similarity between the target variable and the reference variable of each sample log in the first log set; select all sample logs corresponding to the similarity greater than a preset threshold as target sample logs; and use the N parsing rules corresponding to all target sample logs as the N parsing rules corresponding to the log to be accessed.

[0105] As a possible implementation method, the first log set corresponding to the log to be accessed is determined based on the target length and the access type of the log to be accessed, and the first determination parsing rule module 402 is specifically used to: determine the second log set based on the target length; wherein the reference length of the sample log in the second log set is the same as the target length; in the second log set, the sample log with the same access type as the log to be accessed is used as the first log set.

[0106] As a possible implementation method, the similarity between the target variable and the reference variables of each sample log in the first log set is calculated, and the first determination parsing rule module 402 is specifically used to: determine the target variable sequence of the log to be accessed based on the arrangement order of the target variable in the log to be accessed; determine the reference variable sequence of each sample log in the first log set based on the arrangement order of the reference variables of each sample log in the each sample log; use the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log.

[0107] As a possible implementation method, the similarity between the target variable sequence and the reference variable sequence of each sample log is used as the similarity between the target variable and the reference variable of each sample log. The first determination parsing rule module 402 is specifically used to: respectively calculate the longest common subsequence between the target variable sequence and the reference variable sequence of each sample log; and use the length of the longest common subsequence calculated respectively as the similarity between the target variable and the reference variable of each sample log.

[0108] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. The electronic device can realize the functions of the aforementioned device for determining parsing rules, referring to Figure 5 , the electronic device includes:

[0109] At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. Figure 5 The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 may also be referred to as a controller, without limitation to the name.

[0110] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the above-mentioned method for determining parsing rules by executing the instructions stored in the memory 502. The processor 501 can implement Figure 4 The functions of each module in the device / system shown.

[0111] Among them, the processor 501 is the control center of the device / system, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device / system as a whole by running or executing instructions stored in the memory 502 and calling data stored in the memory 502, various functions of the device / system and processing data.

[0112] In one possible design, processor 501 may include one or more processing units. Processor 501 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into processor 501. In some embodiments, processor 501 and memory 502 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0113] The processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method for determining parsing rules disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0114] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device / system that can realize a storage function, for storing program instructions and / or data.

[0115] By designing and programming the processor 501, the code corresponding to the method for determining the parsing rule described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2The steps of the method for determining parsing rules in the embodiment shown are as follows: How to design and program the processor 501 is a technique well known to those skilled in the art and will not be described in detail here.

[0116] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes the method for determining parsing rules discussed above.

[0117] In some possible implementations, various aspects of the method for determining parsing rules provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on the device, the program code is used to enable the control device to execute the steps of the method for determining parsing rules according to various exemplary implementations of the present application described above in this specification.

[0118] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices / systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for determining parsing rules, characterized in that: The method comprises: Determining the number of target variables of the log to be accessed, and using the number of target variables as the target length of the log to be accessed; wherein the target variables represent the field types in the log to be accessed; Based on the target length, determining a second log set; wherein a reference length of the sample logs in the second log set is the same as the target length; In the second log set, a sample log having the same access type as the log to be accessed is used as a first log set; wherein the first log set includes at least one sample log; Calculating the similarity between the target variable and the reference variable of each sample log in the first log set; Select all sample logs with similarities greater than a preset threshold as target sample logs; The N parsing rules corresponding to all target sample logs are used as the N parsing rules corresponding to the log to be accessed; where N is a positive integer greater than or equal to 1; Using the N parsing rules, respectively parse the log to be accessed to obtain N parsing results; wherein one parsing result includes multiple parsing fields; Among the N parsing results, the parsing rule corresponding to the parsing result containing the most parsing fields is used as the target parsing rule corresponding to the log to be accessed.

2. The method according to claim 1, wherein The calculating the similarity between the target variable and the reference variables of each sample log in the first log set includes: Determining a target variable sequence of the log to be accessed based on an arrangement order of the target variables in the log to be accessed; In the first log set, determining a reference variable sequence of each sample log based on an arrangement order of the reference variables of each sample log in the sample log; The similarity between the target variable sequence and the reference variable sequence of each sample log is used as the similarity between the target variable and the reference variable of each sample log.

3. The method according to claim 2, wherein The using the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log includes: respectively calculating the longest common subsequence between the target variable sequence and the reference variable sequence of each sample log; The lengths of the longest common subsequences calculated respectively are used as the similarities between the target variable and the reference variables of each sample log.

4. A device for determining parsing rules, characterized in that: The device comprises: a target length determination module, determining the number of target variables of the log to be accessed, and using the number of target variables as the target length of the log to be accessed; wherein the target variables represent the field types in the log to be accessed; A first parsing rule determination module is configured to determine a second log set based on the target length; wherein the reference length of the sample logs in the second log set is the same as the target length; in the second log set, sample logs of the same access type as the log to be accessed are used as the first log set; wherein the first log set contains at least one sample log; calculate the similarity between the target variable and the reference variable of each sample log in the first log set; select all sample logs corresponding to similarities greater than a preset threshold as target sample logs; and use N parsing rules corresponding to all target sample logs as N parsing rules corresponding to the log to be accessed; wherein N is a positive integer greater than or equal to 1; Obtaining a parsing result module, using the N parsing rules to parse the log to be accessed, respectively, to obtain N parsing results; wherein one parsing result includes multiple parsing fields; The second parsing rule determination module uses, among the N parsing results, a parsing rule corresponding to a parsing result containing the most parsing fields as a target parsing rule corresponding to the log to be accessed.

5. The device according to claim 4, characterized in that The first determination parsing rule module is specifically used to: determine the first log set corresponding to the log to be accessed based on the target length and the access type of the log to be accessed; wherein the first log set contains at least one sample log; calculate the similarity between the target variable and the reference variable of each sample log in the first log set; select all sample logs corresponding to the similarity greater than a preset threshold as target sample logs; and use the N parsing rules corresponding to all target sample logs as the N parsing rules corresponding to the log to be accessed.

6. The device according to claim 4, characterized in that The first determination and parsing rule module is specifically used to: determine the target variable sequence of the log to be accessed based on the arrangement order of the target variable in the log to be accessed; in the first log set, determine the reference variable sequence of each sample log based on the arrangement order of the reference variables of each sample log in the each sample log; and use the similarity between the target variable sequence and the reference variable sequence of each sample log as the similarity between the target variable and the reference variable of each sample log.

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method steps of any one of claims 1 to 3 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 3 are implemented.

Images