A log parsing method and apparatus

Abstract

This application provides a log parsing method and apparatus, applied in the field of log processing technology. The log parsing method includes: obtaining a string to be parsed corresponding to a log to be parsed; traversing the characters in the string to be parsed and parsing them according to parsing rules until the entire string is parsed; wherein the parsing rules include: if the parsed KEY value does not meet a first target condition, then appending a first append character related to the KEY value to the VALUE value of the previous KV pair, and continuing to parse new KEY values; wherein the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length. Therefore, by appending the first append character related to the parsed KEY value to the VALUE value of the previous KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

Description

Technical Field

[0001] This application relates to the field of log processing technology, and more specifically, to a log parsing method and apparatus. Background Technology

[0002] Network devices, systems, and service programs generate event records called logs during operation. Each log line records the date, time, user, and action description. Log auditing and analysis systems typically collect, parse, and store logs to implement auditing and analysis functions. Currently, the main log formats include three types: Sylog, Traffic log, and WELF.

[0003] In the three log formats mentioned above, most of the log content exists as key-value pairs (KV pairs). Currently, regular expressions are generally used to parse the logs; however, in practical applications, due to the non-standardization of log formats, the accuracy of parsing logs using existing methods is relatively low. Summary of the Invention

[0004] The purpose of this application is to provide a log parsing method and apparatus to solve the technical problem of low accuracy in log parsing in the prior art.

[0005] In a first aspect, embodiments of this application provide a log parsing method, comprising: obtaining a string to be parsed corresponding to a log to be parsed; traversing the characters in the string to be parsed and parsing it according to parsing rules until the entire string to be parsed is parsed; wherein, the parsing rules include: if the parsed KEY value does not meet a first target condition, then appending a first append character related to the KEY value to the VALUE value of the previous KV pair, and continuing to parse new KEY values; wherein, the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length.

[0006] In the above scheme, since the next character after the KEY value in a normal KV pair is a connector and the length of the KEY value is less than the preset length, when the parsed KEY value does not meet the first objective condition, it can be considered that the parsed KEY value is not a normal KEY value, but rather a part of the VALUE value of the previous KV pair. Therefore, by appending the first append character related to the parsed KEY value to the VALUE value of the previous KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0007] In an optional implementation, the parsing rule further includes: if the parsed KEY value satisfies the first target condition, then continue parsing new VALUE values. In the above scheme, since in a normal KV pair, the next character of the KEY value is a connector and the length of the KEY value is less than a preset length, when the parsed KEY value satisfies the first target condition, it can be considered a normal KEY value. Therefore, new VALUE values ​​can continue to be parsed, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0008] In an optional implementation, the parsing rule further includes: if the parsed VALUE value satisfies a second target condition, then continue parsing a new KEY value; wherein the second target condition includes: the previous character of the parsed VALUE value is a front enclosing character, the next character of the parsed VALUE value is a back enclosing character, and the next character of the back enclosing character is a separator, and / or, the second target condition is that the previous character of the parsed VALUE value is not a front enclosing character and the next character of the parsed VALUE value is a separator. In the above scheme, since in a normal KV pair, the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character, and the next character of the back enclosing character is a separator, or the previous character of the VALUE value is not a front enclosing character and the next character of the VALUE value is a separator, when the parsed VALUE value satisfies the second target condition, it can be considered a normal VALUE value. Therefore, the step of determining the KEY value of the next KV pair can be performed, thereby ensuring the accuracy of the VALUE value and improving the accuracy of log parsing.

[0009] In an optional implementation, the parsing rule further includes: if the parsed VALUE value satisfies a third target condition, then a second appended character related to the parsed VALUE value is appended to the VALUE value of the KV pair; wherein the third target condition includes: the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is not a separator, and / or, the previous character of the VALUE value is a front enclosing character and the next character of the VALUE value is not a back enclosing character. In the above scheme, since in a normal KV pair, the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is a separator, or the previous character of the VALUE value is not a front enclosing character and the next character of the VALUE value is a separator, therefore, when the parsed VALUE value satisfies the third target condition, it can be considered that the parsed VALUE value is not a normal VALUE value. Therefore, by adding a second additional payment associated with the parsed VALUE value to the VALUE value of the KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0010] In an optional implementation, the parsing rule further includes: if the parsed KEY value meets a fourth target condition, then the parsed KEY value is replaced using a replacement value; wherein the fourth target condition is that the parsed KEY value needs to be replaced. In the above scheme, when it is necessary to replace the parsed KEY value, it can be replaced according to a pre-set replacement value, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0011] In an optional implementation, the connector, delimiter, front encloser, and back encloser each include at least one character. In the above scheme, when the connector, delimiter, front encloser, or back encloser all include multiple characters, the string can be parsed according to their specific character characteristics, thereby ensuring the accuracy of the parsing results and improving the accuracy of log parsing.

[0012] Secondly, embodiments of this application provide a log parsing device, comprising: an acquisition module for acquiring a string to be parsed corresponding to a log to be parsed; and a traversal module for traversing the characters in the string to be parsed and parsing them according to parsing rules until the entire string to be parsed is parsed; wherein the parsing rules include: if the parsed KEY value does not meet a first target condition, then a first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the new KEY value is parsed; wherein the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length.

[0013] In the above scheme, since the next character after the KEY value in a normal KV pair is a connector and the length of the KEY value is less than the preset length, when the parsed KEY value does not meet the first objective condition, it can be considered that the parsed KEY value is not a normal KEY value, but rather a part of the VALUE value of the previous KV pair. Therefore, by appending the first append character related to the parsed KEY value to the VALUE value of the previous KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0014] In an optional implementation, the parsing rule further includes: if the parsed KEY value satisfies the first target condition, then continue parsing new VALUE values. In the above scheme, since in a normal KV pair, the next character of the KEY value is a connector and the length of the KEY value is less than a preset length, when the parsed KEY value satisfies the first target condition, it can be considered a normal KEY value. Therefore, new VALUE values ​​can continue to be parsed, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0015] In an optional implementation, the parsing rule further includes: if the parsed VALUE value satisfies a second target condition, then continue parsing a new KEY value; wherein the second target condition includes: the previous character of the parsed VALUE value is a front enclosing character, the next character of the parsed VALUE value is a back enclosing character, and the next character of the back enclosing character is a separator, and / or, the second target condition is that the previous character of the parsed VALUE value is not a front enclosing character and the next character of the parsed VALUE value is a separator. In the above scheme, since in a normal KV pair, the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character, and the next character of the back enclosing character is a separator, or the previous character of the VALUE value is not a front enclosing character and the next character of the VALUE value is a separator, when the parsed VALUE value satisfies the second target condition, it can be considered a normal VALUE value. Therefore, the step of determining the KEY value of the next KV pair can be performed, thereby ensuring the accuracy of the VALUE value and improving the accuracy of log parsing.

[0016] In an optional implementation, the parsing rule further includes: if the parsed VALUE value satisfies a third target condition, then a second appended character related to the parsed VALUE value is appended to the VALUE value of the KV pair; wherein the third target condition includes: the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is not a separator, and / or, the previous character of the VALUE value is a front enclosing character and the next character of the VALUE value is not a back enclosing character. In the above scheme, since in a normal KV pair, the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is a separator, or the previous character of the VALUE value is not a front enclosing character and the next character of the VALUE value is a separator, therefore, when the parsed VALUE value satisfies the third target condition, it can be considered that the parsed VALUE value is not a normal VALUE value. Therefore, by adding a second additional payment associated with the parsed VALUE value to the VALUE value of the KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0017] In an optional implementation, the parsing rule further includes: if the parsed KEY value meets a fourth target condition, then the parsed KEY value is replaced using a replacement value; wherein the fourth target condition is that the parsed KEY value needs to be replaced. In the above scheme, when it is necessary to replace the parsed KEY value, it can be replaced according to a pre-set replacement value, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0018] In an optional implementation, the connector, delimiter, front encloser, and back encloser each include at least one character. In the above scheme, when the connector, delimiter, front encloser, or back encloser all include multiple characters, the string can be parsed according to their specific character characteristics, thereby ensuring the accuracy of the parsing results and improving the accuracy of log parsing.

[0019] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which are read and executed by a processor to perform the log parsing method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus; the processor and the memory communicate with each other via the bus; the memory stores computer program instructions that can be executed by the processor, and the processor can execute the log parsing method as described in the first aspect by calling the computer program instructions.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium that stores computer program instructions, which, when executed by a computer, cause the computer to perform the log parsing method as described in the first aspect.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, embodiments of this application are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of a Syslog format log provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of a Traffic log format provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of a WELF format log provided in an embodiment of this application;

[0027] Figure 4 A flowchart illustrating a log parsing method provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram illustrating another log parsing method provided in an embodiment of this application;

[0029] Figure 6 A schematic diagram of a string to be parsed provided for an embodiment of this application;

[0030] Figure 7 A structural block diagram of a log parsing device provided in an embodiment of this application;

[0031] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] Currently, the main log formats include three types: Sylog, Traffic log, and WELF.

[0033] The Syslog format is defined by RFC 3164. This protocol was originally defined and implemented by the Transmission Control Protocol / Internet Protocol (TCP / IP) group at California Berkeley for transmitting events and messages from devices over a network. Syslog is supported by Unix, Windows, and most mainstream network devices. For example, its format is as follows: Figure 1 As shown.

[0034] Traffic logs are traffic-based logs kept by devices, compensating for the lack of traffic information in Syslog. Information is generated after each access to the network device ends, and together with Syslog-formatted logs, it allows for monitoring of device traffic. For example, its format is as follows: Figure 2 As shown.

[0035] WELF, developed by WebTrends, is a log format supported by many firewalls. WELF format logs can be stored directly as files or transmitted to log servers via systems like Syslog. For example, its format is as follows: Figure 3 As shown.

[0036] As can be seen from the three log formats above, most of the log content exists in the form of key-value pairs, that is, in the form of "KEY=VALUE" or "KEY="VALUE"". In these formats, "=" is the connector, "" is the leading and trailing encloser, and the space between two key-value pairs is the separator.

[0037] The applicant discovered that, in practical applications, due to non-standard log formats, the key-value pairs in the logs may not necessarily exist in the form described above. For example, the key value may contain a concatenation character; or the value may contain a leading and trailing enclosing character, a delimiter, and a concatenation character; or the concatenation character, delimiter, leading and trailing enclosing character may contain multiple characters, etc.

[0038] If the logs are parsed using existing methods, inaccurate key-value pairs will be obtained. Therefore, this application provides a log parsing method that can still parse high-accuracy key-value pairs even when the log format is not standardized.

[0039] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0040] Please refer to Figure 4 , Figure 4 A flowchart of a log parsing method provided in this application embodiment, the log parsing method may include the following steps:

[0041] Step S401: Obtain the string to be parsed corresponding to the log to be parsed.

[0042] Step S402: Traverse the characters in the string to be parsed and parse them according to the parsing rules until the entire string is parsed.

[0043] Specifically, in step S401 above, the log to be parsed refers to the log that needs to be parsed. The specific implementation method of the log to be parsed in this embodiment is not specifically limited; those skilled in the art can make appropriate adjustments based on actual circumstances. For example, the log to be parsed can be in Sylog format, Traffic log format, or ELF format, etc.; or, the log to be parsed can be generated by network devices, systems, or service programs, etc.

[0044] The string to be parsed is the string in the aforementioned log to be parsed. In this embodiment of the application, the string to be parsed includes multiple key-value pairs. Therefore, by parsing the string to be parsed, the corresponding multiple key-value pairs can be obtained.

[0045] It should be noted that the embodiments of this application do not impose specific limitations on the specific implementation of obtaining the string to be parsed. Those skilled in the art can make appropriate adjustments according to the actual situation. For example, the string to be parsed can be obtained by reading the locally stored log; or, the string to be parsed can be received from an external device, etc.

[0046] In step S402 above, the characters in the string to be parsed can be traversed sequentially, and the string to be parsed can be processed according to the parsing rules until all characters in the string to be parsed are parsed.

[0047] According to the parsing rules, the key value of a key-value pair can be extracted from the string to be parsed first. If the key value satisfies the above parsing rules, it can be identified as the key value of that key-value pair, and the value of the value in the key-value pair can be extracted from the string to be parsed. If the value of the value satisfies the above parsing rules, it can be identified as the value of that key-value pair. After determining the key value and value of the key-value pair, the key value of the next key-value pair can be parsed. This process is repeated until all characters have been parsed.

[0048] The parsing rules provided in the embodiments of this application will be described in detail below.

[0049] The parsing rules include: if the parsed KEY value does not meet the first target condition, the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the parsing of new KEY values ​​continues.

[0050] The first objective condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than the preset length. After parsing the KEY value of a KV pair from the string to be parsed, the parsed KEY value cannot be directly identified as the KEY value of the KV pair; instead, it can be determined whether the parsed KEY value satisfies the first objective condition.

[0051] Since in a normal key-value pair, the next character after the key is a connector and the length of the key is less than the preset length, if the parsed key does not meet the first objective condition mentioned above, it can be considered that the parsed key cannot be used as a key, and the previous value is not a complete value. Therefore, the first append character related to the key can be appended to the value of the previous key-value pair, thereby increasing the length of the previous value and making it a complete value.

[0052] It should be noted that the embodiments of this application do not specifically limit the specific implementation of the first appended character. Those skilled in the art can make appropriate adjustments according to the actual situation. For example, the first appended character can be the parsed KEY value; or, the first appended character can include the parsed KEY value and some characters before or after the parsed KEY value.

[0053] Furthermore, this application does not impose specific limitations on the specific implementation of the above-mentioned preset length. Those skilled in the art can make appropriate adjustments according to the actual situation. For example, the preset length can be 10 characters, 15 characters, etc.

[0054] Next, since the previously parsed KEY value has already been appended to the VALUE value of the previous KV pair, the KEY value of the next KV pair can be parsed again.

[0055] For example, suppose the string to be parsed includes: key1 = value1 value2". The KEY value in the previous KV pair is "key1" and the VALUE value is "value1". The parsed KEY value is "value2", and its next character is "", so the first target condition mentioned above is not met. The VALUE value in the previous KV pair is updated to "value1 value2".

[0056] For example, suppose the string to be parsed includes: key1=value1 interference interference interference interference interference interference=, the KEY value in the previous KV pair is "key1" and the VALUE value is "value1"; the parsed KEY value is "interference interference interference interference interference interference", which is longer than the preset length, so it does not meet the first target condition mentioned above; the VALUE value in the previous KV pair is updated to "value1 interference interference interference interference interference interference=".

[0057] In the above scheme, since the next character after the KEY value in a normal KV pair is a connector and the length of the KEY value is less than the preset length, when the parsed KEY value does not meet the first objective condition, it can be considered that the parsed KEY value is not a normal KEY value, but rather a part of the VALUE value of the previous KV pair. Therefore, by appending the first append character related to the parsed KEY value to the VALUE value of the previous KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0058] Furthermore, based on the above embodiments, the parsing rules may also include: if the parsed KEY value satisfies the first target condition, then continue parsing a new VALUE value.

[0059] Specifically, since the next character after the KEY value in a normal KV pair is a connector and the length of the KEY value is less than the preset length, if the parsed KEY value meets the first objective condition mentioned above, it can be considered as a KEY value and the VALUE value in the KV pair can be parsed.

[0060] For example, suppose the string to be parsed includes: key1 = value1 key2 = value2, the KEY value of the previous KV pair is "key1" and the VALUE value is "value1"; the parsed KEY value is "key2", the next character is "=" and its length is less than the preset length, so the first target condition is met, the KEY value of the KV pair is determined to be "key2", and the VALUE value of the KV pair is parsed.

[0061] In the above scheme, since the next character after the key value in a normal key-value pair is a connector and the length of the key value is less than the preset length, a key value that meets the first target condition can be considered a normal key value. Therefore, new value parsing can continue, thus ensuring the accuracy of the key value and improving the accuracy of log parsing.

[0062] Furthermore, based on the above embodiments, the parsing rules may also include: if the parsed VALUE value satisfies the second target condition, then continue parsing a new KEY value.

[0063] Specifically, the second objective condition includes: the previous character of the parsed VALUE value is a front enclosing character, the next character of the parsed VALUE value is a back enclosing character and the next character of the back enclosing character is a separator, and / or, the second objective condition is that the previous character of the parsed VALUE value is not a front enclosing character and the next character of the parsed VALUE value is a separator.

[0064] After parsing the VALUE value of a key-value pair from the string to be parsed, the parsed VALUE value cannot be directly determined as the VALUE value of the key-value pair; instead, it can be determined whether the parsed VALUE value satisfies the second objective condition.

[0065] Since in a normal key-value pair, the character preceding the value is the front enclosing character, the character following the value is the back enclosing character, and the character following the back enclosing character is the separator, or the character preceding the value is not the front enclosing character and the character following the value is the separator, if the parsed value satisfies the second objective condition mentioned above, it can be considered as a value and can be used to parse the key value in the next key-value pair.

[0066] For example, suppose the string to be parsed includes: key1 = value1 and key2 = value2. The key value in the determined KV pair is "key1", and the parsed value is "value1". The parsed key value is "key2", with the previous character being "" and the next character being "", and the next character being a space. Therefore, the second objective condition mentioned above is satisfied, and the value of the KV pair is determined to be "value1". Then, the key value in the next KV pair is parsed.

[0067] In the above scheme, since in a normal key-value pair, the character preceding the value is the leading border, the character following the value is the trailing border, and the character following the trailing border is the separator, or the character preceding the value is not the leading border and the character following the value is the separator, the parsed value can be considered a normal value if it meets the second objective condition. Therefore, the step of determining the key value of the next key-value pair can be performed to ensure the accuracy of the value and improve the accuracy of log parsing.

[0068] Furthermore, based on the above embodiments, the parsing rules may also include: if the parsed VALUE value satisfies the third target condition, then the second append character related to the parsed VALUE value is appended to the VALUE value of the KV pair.

[0069] Specifically, the third target condition includes: the character preceding the VALUE value is a front enclosing character, the character following the VALUE value is a back enclosing character and the character following the back enclosing character is not a separator, and / or, the character preceding the VALUE value is a front enclosing character and the character following the VALUE value is not a back enclosing character.

[0070] After parsing the VALUE value of a key-value pair from the string to be parsed, the parsed VALUE value cannot be directly determined as the VALUE value of the key-value pair; instead, the subsequent characters can be parsed. If the VALUE value is still being parsed, it can be determined whether the parsed VALUE value satisfies the third objective condition.

[0071] Since in a normal key-value pair, the character preceding the value is the front enclosing character, the character following the value is the back enclosing character, and the character following the back enclosing character is the separator, or the character preceding the value is not the front enclosing character and the character following the value is the separator, if the parsed value satisfies the third objective condition mentioned above, then the previous value can be considered not to be a complete value, and the second append character related to the parsed value can be appended to the value of the key-value pair.

[0072] It should be noted that the embodiments of this application do not specifically limit the specific implementation of the above-mentioned second appended character. Those skilled in the art can make appropriate adjustments according to the actual situation. For example, the second appended character can be the parsed VALUE value; or, the second appended character can include the parsed VALUE value and some characters before or after the parsed VALUE value.

[0073] In the above scheme, since in a normal key-value pair, the character preceding the value is the preceding enclosing character, the character following the value is the following enclosing character, and the character following the following enclosing character is the separator, or the character preceding the value is not the preceding enclosing character and the character following the value is the separator, when the parsed value satisfies the third objective condition, it can be considered that the parsed value is not a normal value. Therefore, by appending the second append payment related to the parsed value to the value of the key-value pair, the accuracy of the value is ensured, thereby improving the accuracy of log parsing.

[0074] Furthermore, based on the above embodiments, the parsing rules may also include: if the parsed KEY value satisfies the fourth target condition, then the parsed KEY value is replaced using a replacement value.

[0075] Specifically, the fourth objective condition is that the parsed KEY value needs to be replaced. It should be noted that the replacement value can be pre-set, and this embodiment does not impose specific limitations on it.

[0076] For example, suppose the string to be parsed includes: key1 = key5 = value1. The KEY value in the parsed KV pair is "key1". This KEY value satisfies the third objective condition mentioned above, and the replacement value is key1 = key5. Therefore, the KEY value in the KV pair is updated to "key1 = key5".

[0077] In the above scheme, when it is necessary to replace the parsed KEY value, it can be replaced according to the pre-set replacement value, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0078] Furthermore, based on the above embodiments, the connector, separator, front enclosing character, and back enclosing character all include at least one character.

[0079] Specifically, in one implementation, the connector can be “=", the pre-enclosing character can be “"”, the post-enclosing character can be “"”, and the separator can be a space; in another implementation, the connector, separator, pre-enclosing character, and post-enclosing character all include at least one character.

[0080] It should be noted that in the second embodiment described above, the specific implementation of the connector, separator, front enclosing character, and back enclosing character can be set in advance, and this application embodiment does not impose specific limitations on this.

[0081] In the above scheme, when the connector, delimiter, front enclosing character, or back enclosing character all contain multiple characters, the string can be parsed according to its specific character information, thereby ensuring the accuracy of the parsing results and improving the accuracy of log parsing.

[0082] Based on the above implementation methods, this application also provides another log parsing method. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of another log parsing method provided in an embodiment of this application. The log parsing method includes the following steps:

[0083] Step 1) Initialize the variables by setting "Start processing KEY" to true, "VALUE has a leading character" to true, "KEY value" to empty, and "VALUE value" to empty; then start iterating through the characters of the log string.

[0084] Step 2), determine if "Start processing KEY" is true.

[0085] Step 3), if "Start processing KEY" is true, there are four possible scenarios:

[0086] Scenario 1: If the next character after the parsed "KEY value" is a concatenation character, then set "Start processing KEY" to false, replace the "KEY value" to be converted, check if the "VALUE value" has a leading enclosing character and update the value of "VALUE has a leading enclosing character".

[0087] Understandably, in scenario one above, the parsed KEY value satisfies the first objective condition, so in this case, we can continue parsing new VALUE values. Furthermore, the parsed KEY value may satisfy the third objective condition, so we can use a replacement value to replace the parsed KEY value.

[0088] Scenario 2: If the next character after the parsed "KEY value" is a separator, then append the first append character related to the "KEY value" to the "VALUE value" of the previous KV pair, store the KV pair, set the "KEY value" to empty, and set the "VALUE value" to empty.

[0089] Understandably, in scenario two above, the parsed KEY value does not meet the first target condition. Therefore, in this case, the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the parsing of the new KEY value continues.

[0090] Case 3: If the parsed "KEY value" ends at the end of the string, then append the "KEY value" to the "VALUE value" of the previous KV pair and store the KV pair.

[0091] Understandably, in scenario three above, the parsed KEY value does not meet the first target condition. Therefore, in this case, the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and parsing of new KEY values ​​continues. Encountering the end of the string indicates that the string to be parsed has been completely parsed.

[0092] Case 4: If the next character after the parsed "KEY value" is another character, then append that character to the "KEY value".

[0093] It is understandable that in the above case four, it can be considered as the process of updating the parsed KEY value; during the process of traversing the string to be parsed, characters that are not connectors or delimiters are all determined to be part of the parsed KEY value.

[0094] Step 4): If “Start processing KEY” is false, then determine whether “VALUE has a leading enclosing character” is true.

[0095] Step 5): If "VALUE has a leading enclosing character" is true, then determine whether the next character of "VALUE value" is a trailing enclosing character.

[0096] Step 6): If the next character of "VALUE value" is a closing character, there are four possible scenarios:

[0097] Case 1: If the end of the string is after the closing encloser, then the string traversal ends.

[0098] Scenario 2: If the character following the closing enclosing character is a delimiter, then set "Start processing KEY" to true, set "VALUE has a closing enclosing character" to false, store the KV pair, set "KEY value" to empty, and set "VALUE value" to empty.

[0099] Understandably, in scenario two above, the parsed VALUE value satisfies the second objective condition, so in this case, we can continue to parse new KEY values.

[0100] Case 3: If the character following the closing enclosing character is another character, then the character is appended to the "VALUE value".

[0101] It is understandable that in the above situation three, it can be considered as the process of updating the parsed VALUE value; during the process of traversing the string to be parsed, when the previous character of the parsed VALUE value is the front enclosing character and the next character of the parsed VALUE value is the back enclosing character, the character that is not a delimiter after the back enclosing character is determined as part of the parsed VALUE value.

[0102] Step 7) If the next character after “VALUE” is not a closing character, then append the character after “VALUE”.

[0103] It is understandable that in step 7) above, the parsed VALUE value satisfies the third objective condition. Therefore, in this case, the second append character related to the parsed VALUE value is appended to the VALUE value of the KV pair.

[0104] Step 8): If “VALUE has a leading enclosing character” is false and “KEY value” is too long, then append “KEY value” to the previous KV pair.

[0105] It is understandable that in step 8) above, the parsed KEY value does not meet the first target condition. Therefore, in this case, the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the new KEY value is parsed.

[0106] Step 9): If “VALUE has a leading enclosing character” is false, then determine whether the next character of “VALUE value” is a delimiter.

[0107] Step 10): If the next character after “VALUE” is a separator, then set “Start processing KEY” to true, store the KV pair, set “KEY” to empty, and set “VALUE” to empty.

[0108] Step 11): If the next character after “VALUE” is not a separator, then append the character to “VALUE”.

[0109] The following examples illustrate four abnormal situations that the log parsing method provided in this application can handle. Please refer to... Figure 6 , Figure 6 This is a schematic diagram of a string to be parsed, provided as an embodiment of this application.

[0110] For cases where the KEY value contains a connector:

[0111] like Figure 6 As shown in the figure, “key=5=value5”, the log parsing method provided in this application embodiment is used to traverse characters from left to right. When the first “=" is encountered, the KEY value is set to “key”. At the same time, it is determined whether the KEY value satisfies the fourth target condition. If it does, the KEY value is set to “key=5”. At the same time, the position of the difference between the lengths of the two key values ​​is skipped and the characters are traversed (i.e., the “value” character is skipped) until a space is encountered, and the VALUE value is “value5”.

[0112] It should be noted that in practical applications, for example, if the characters to be parsed include "HTTP / S response code / 404", it should be parsed as KEY value "HTTP / S response code" and VALUE value "404".

[0113] For cases where the value contains both leading and trailing enclosing characters:

[0114] like Figure 6 As shown in the example "key4 = "value"interference"4", using the log parsing method provided in this application, characters are traversed from left to right. When an "=" is encountered, the KEY value is set to "key4". When the first "" is encountered, "VALUE has a leading enclosing character" is set to true. Before the second "" is encountered, the VALUE value is set to "value". When the second "" is encountered, it is determined that the next character is not a delimiter, and the VALUE value is set to "value". When the second space is encountered, the VALUE value is set to "value"interference". The next KV pair is processed. When the third space is encountered, it means that the KEY value is "4" at this time, but the KV pair is incomplete. It is determined that the KEY value ends with the "ending enclosing character". After the VALUE value of the previous KV pair, "", space, and KEY value are appended in sequence (the "ending enclosing character" is deleted), that is, the VALUE value is set to "value"interference"4.

[0115] It should be noted that in practical applications, for example, the "message" field includes both preceding and following enclosing characters.

[0116] For cases where the value contains a separator:

[0117] like Figure 6 As shown in “key2 = value2”, using the log parsing method provided in this application embodiment, the characters are traversed from left to right. When the first “=" is encountered, the KEY value is set to “key2”. When a space is encountered, the VALUE value is set to “value”. The next KV pair is processed. When a space is encountered again, it means that the KEY value is “2” but the KV pair is incomplete. The KEY value is then appended to the VALUE value of the previous KV pair, that is, the VALUE value is set to “value2”.

[0118] For cases where the value contains a connector:

[0119] like Figure 6 As shown in the figure “key6=value6 interference interference interference interference interference interference==", using the log parsing method provided in this application embodiment, the characters are traversed from left to right. When a space is encountered, the KEY value is set to “key6” and the VALUE value is “value6”. When the second “=" is encountered, the KEY value is set to “interference interference interference interference interference interference”. When the third “=" is encountered, it is determined that the KEY value is too long. Then, a space, the KEY value, “=", and the “=" encountered this time are appended after the VALUE value of the previous KV pair. That is, the KEY value is set to “key=key6” and the VALUE value is “value6 interference interference interference interference interference interference==". The process ends when the end of the string is encountered.

[0120] It should be noted that in practical applications, for example, the "message" field may contain spaces and base64 characters.

[0121] Please refer to Figure 7 , Figure 7 This application provides a structural block diagram of a log parsing device 700, comprising: an acquisition module 701 for acquiring a string to be parsed corresponding to a log to be parsed; and a traversal module 702 for traversing the characters in the string to be parsed and parsing them according to parsing rules until the entire string to be parsed is parsed. The parsing rules include: if the parsed KEY value does not meet a first target condition, then a first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the parsing of new KEY values ​​continues. The first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length.

[0122] In the above scheme, since the next character after the KEY value in a normal KV pair is a connector and the length of the KEY value is less than the preset length, when the parsed KEY value does not meet the first objective condition, it can be considered that the parsed KEY value is not a normal KEY value, but rather a part of the VALUE value of the previous KV pair. Therefore, by appending the first append character related to the parsed KEY value to the VALUE value of the previous KV pair, the accuracy of the VALUE value is ensured, thereby improving the accuracy of log parsing.

[0123] Furthermore, based on the above embodiments, the parsing rule also includes: if the parsed KEY value satisfies the first target condition, then continue to parse a new VALUE value.

[0124] In the above scheme, since the next character after the key value in a normal key-value pair is a connector and the length of the key value is less than the preset length, a key value that meets the first target condition can be considered a normal key value. Therefore, new value parsing can continue, thus ensuring the accuracy of the key value and improving the accuracy of log parsing.

[0125] Furthermore, based on the above embodiments, the parsing rule further includes: if the parsed VALUE value satisfies the second target condition, then continue parsing a new KEY value; wherein, the second target condition includes: the previous character of the parsed VALUE value is a front enclosing character, the next character of the parsed VALUE value is a back enclosing character and the next character of the back enclosing character is a separator, and / or, the second target condition is that the previous character of the parsed VALUE value is not a front enclosing character and the next character of the parsed VALUE value is a separator.

[0126] In the above scheme, since in a normal key-value pair, the character preceding the value is the leading border, the character following the value is the trailing border, and the character following the trailing border is the separator, or the character preceding the value is not the leading border and the character following the value is the separator, the parsed value can be considered a normal value if it meets the second objective condition. Therefore, the step of determining the key value of the next key-value pair can be performed to ensure the accuracy of the value and improve the accuracy of log parsing.

[0127] Furthermore, based on the above embodiments, the parsing rule further includes: if the parsed VALUE value satisfies the third target condition, then the second append character related to the parsed VALUE value is appended to the VALUE value of the KV pair; wherein, the third target condition includes: the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is not a separator, and / or, the previous character of the VALUE value is a front enclosing character and the next character of the VALUE value is not a back enclosing character.

[0128] In the above scheme, since in a normal key-value pair, the character preceding the value is the preceding enclosing character, the character following the value is the following enclosing character, and the character following the following enclosing character is the separator, or the character preceding the value is not the preceding enclosing character and the character following the value is the separator, when the parsed value satisfies the third objective condition, it can be considered that the parsed value is not a normal value. Therefore, by appending the second append payment related to the parsed value to the value of the key-value pair, the accuracy of the value is ensured, thereby improving the accuracy of log parsing.

[0129] Furthermore, based on the above embodiments, the parsing rule further includes: if the parsed KEY value satisfies the fourth target condition, then the parsed KEY value is replaced using a replacement value; wherein, the fourth target condition is that the parsed KEY value needs to be replaced.

[0130] In the above scheme, when it is necessary to replace the parsed KEY value, it can be replaced according to the pre-set replacement value, thereby ensuring the accuracy of the KEY value and improving the accuracy of log parsing.

[0131] Furthermore, based on the above embodiments, the connector, separator, front enclosing character, and back enclosing character all include at least one character.

[0132] In the above scheme, when the connector, delimiter, front enclosing character, or back enclosing character all contain multiple characters, the string can be parsed according to its specific character information, thereby ensuring the accuracy of the parsing results and improving the accuracy of log parsing.

[0133] Please refer to Figure 8 , Figure 8This application provides a structural block diagram of an electronic device 800, which includes at least one processor 801, at least one communication interface 802, at least one memory 803, and at least one communication bus 804. The communication bus 804 enables direct communication between these components, the communication interface 802 facilitates signaling or data communication with other node devices, and the memory 803 stores machine-readable instructions executable by the processor 801. When the electronic device 800 is running, the processor 801 communicates with the memory 803 via the communication bus 804, and the aforementioned log parsing method is executed when the machine-readable instructions are invoked by the processor 801.

[0134] For example, in this embodiment of the application, the processor 801 reads a computer program from the memory 803 via the communication bus 804 and executes the computer program to implement the following method: Step S401: Obtain the string to be parsed corresponding to the log to be parsed. Step S402: Traverse the characters in the string to be parsed and parse it according to the parsing rules until the entire string to be parsed is parsed.

[0135] The processor 801 may include one or more, and may be an integrated circuit chip with signal processing capabilities. The processor 801 may be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; or it may be a special-purpose processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 801, some may be general-purpose processors, and others may be special-purpose processors.

[0136] The memory 803 includes one or more, which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0137] Understandable. Figure 8 The structure shown is for illustrative purposes only; the electronic device 800 may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown. Figure 8 The components shown can be implemented using hardware, software, or a combination thereof. In the embodiments of this application, electronic device 800 can be, but is not limited to, physical devices such as desktop computers, laptops, smartphones, smart wearable devices, and in-vehicle devices, or virtual devices such as virtual machines. Furthermore, electronic device 800 is not necessarily a single device; it can be a combination of multiple devices, such as a server cluster, etc.

[0138] This application also provides a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes computer program instructions. When the computer program instructions are executed by a computer, the computer can perform the steps of the log parsing method described in the above embodiments, such as: obtaining the string to be parsed corresponding to the log to be parsed; traversing the characters in the string to be parsed and parsing it according to the parsing rules until the entire string to be parsed is parsed; wherein, the parsing rules include: if the parsed KEY value does not meet the first target condition, then appending the first append character related to the KEY value to the VALUE value of the previous KV pair, and continuing to parse new KEY values; wherein, the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length.

[0139] This application also provides a computer-readable storage medium that stores computer program instructions. When the computer program instructions are executed by a computer, the computer performs the log parsing method described in the foregoing method embodiments.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0141] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0143] It should be noted that if the function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0145] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log parsing method characterized by, include: Retrieve the string to be parsed corresponding to the log to be parsed; Iterate through the characters in the string to be parsed and parse them according to the parsing rules until the entire string is parsed. The parsing rules include: If the parsed KEY value does not meet the first target condition, then the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the parsing of new KEY values ​​continues; wherein, the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length, the first append character is the parsed KEY value, or, the first append character includes the parsed KEY value and some characters before or after the parsed KEY value.

2. The log analysis method according to claim 1, characterized in that, The parsing rules also include: If the parsed KEY value satisfies the first target condition, then continue parsing a new VALUE value. 3.The log analysis method according to claim 1, characterized in that, The parsing rules also include: If the parsed VALUE value satisfies the second objective condition, then continue parsing the new KEY value; The second target condition includes: the previous character of the parsed VALUE value is a front enclosing character, the next character of the parsed VALUE value is a back enclosing character, and the next character of the back enclosing character is a separator; or, the second target condition is that the previous character of the parsed VALUE value is not a front enclosing character and the next character of the parsed VALUE value is a separator.

4. The log analysis method of claim 1, wherein, The parsing rules also include: If the parsed VALUE value satisfies the third target condition, then the second append character related to the parsed VALUE value is appended to the VALUE value of the KV pair; The third target condition includes: the previous character of the VALUE value is a front enclosing character, the next character of the VALUE value is a back enclosing character and the next character of the back enclosing character is not a separator, or the previous character of the VALUE value is a front enclosing character and the next character of the VALUE value is not a back enclosing character.

5. The log parsing method according to claim 1, characterized in that, The parsing rules also include: If the parsed KEY value meets the fourth objective condition, then the parsed KEY value is replaced using a replacement value; wherein, the fourth objective condition is that the parsed KEY value needs to be replaced.

6. The log parsing method according to any one of claims 1-5, characterized in that, Connectors, separators, front enclosing characters, and back enclosing characters all include at least one character.

7. A log parsing device, characterized in that, include: The acquisition module is used to obtain the string to be parsed corresponding to the log to be parsed; The traversal module is used to traverse the characters in the string to be parsed and parse them according to the parsing rules until the entire string to be parsed is parsed. The parsing rules include: If the parsed KEY value does not meet the first target condition, then the first append character related to the KEY value is appended to the VALUE value of the previous KV pair, and the parsing of new KEY values ​​continues; wherein, the first target condition is that the next character of the parsed KEY value is a connector and the length of the parsed KEY value is less than a preset length, the first append character is the parsed KEY value, or, the first append character includes the parsed KEY value and some characters before or after the parsed KEY value.

8. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the log parsing method as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor can execute the log parsing method as described in any one of claims 1-6 by calling the computer program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a computer, cause the computer to perform the log parsing method as described in any one of claims 1-6.

Images