Automatic processing method, device, equipment and medium for non-standardized log structure

Abstract

The present invention discloses a method, device, equipment and medium for automatically processing a non-standardized log structure. The method comprises: identifying a target log to be processed, and when it is determined that the target log is a non-standardized log structure according to an automatic parsing rule, performing feature abstraction and aggregation to obtain an abstract key-value pair string; identifying a target separator in the abstract key-value pair string; identifying a missing character position and a missing character type in the abstract key-value pair string according to the target separator, and filling in a supplementary character matching the missing character type at the matching missing character position in the target log to obtain a standardized log structure. The technical solution of the embodiment of the present invention obtains a standardized log structure by identifying missing characters and automatically filling in the missing characters, and avoids the problem of low efficiency in the prior art that only regular expressions can be used to parse non-standardized log structures.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, equipment and medium for automatically processing a non-standardized log structure. Background Art

[0002] In order to conduct accurate statistical analysis on log data, the log management system uses key information annotation and preprocessing methods to format the key clues and key features in the log into fields that are convenient for statistical calculation. This process is called data parsing. Among them, log data usually consists of "regular log content" and "message structure content".

[0003] However, in the actual output process of log data, due to operational errors, log content being too long, and log content exceeding the sample range set by the parsing rules, the "message structure" content is easily truncated, resulting in missing content and becoming a non-standardized structure, resulting in incomplete data coverage and inaccurate parsing results.

[0004] Therefore, there is an urgent need for a method for processing a non-standardized log structure to obtain a standardized log structure, thereby obtaining accurate log data parsing results. Summary of the invention

[0005] Based on this, the present invention provides a method, device, equipment and medium for automatically processing a non-standardized log structure, so as to solve the problem that the content of the message structure may be non-standardized during the actual output of log data.

[0006] In a first aspect, an embodiment of the present invention provides a method for automatically processing a non-standardized log structure, the method comprising:

[0007] Identify the target log to be processed, and when it is determined according to the automatic parsing rule that the target log is a non-standardized log structure, perform feature abstraction and aggregation to obtain an abstract key-value pair string;

[0008] Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair;

[0009] In the abstract key-value pair string, identifying a target separator, wherein the target separator includes: a key name key value separator, an object group separator, a key value pair separator, and a key name key value identifier;

[0010] The character missing position and the character missing type are identified in the abstract key-value pair string according to the target delimiter, and supplementary characters matching the character missing type are filled in the matching character missing position in the target log to obtain a standardized log structure.

[0011] In a second aspect, an embodiment of the present invention further provides a non-standardized log structure automatic processing device, the device comprising:

[0012] An abstract key-value pair string determination module is used to identify a target log to be processed, and when the target log is determined to be a non-standardized log structure according to an automatic parsing rule, perform feature abstraction and aggregation to obtain an abstract key-value pair string;

[0013] Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair;

[0014] A target delimiter determination module is used to identify a target delimiter in the abstract key-value pair string, wherein the target delimiter includes: a key name key value delimiter, an object group delimiter, a key value pair delimiter and a key name key value identifier;

[0015] The standardized log structure generation module is used to identify the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and fill the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure.

[0016] In a third aspect, an embodiment of the present invention further provides a device, the device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for automatically processing a non-standardized log structure described in any embodiment of the present invention.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for automatically processing a non-standardized log structure as described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention can automatically determine the non-standardized log structure, and obtain the standardized log structure by identifying missing characters and automatically filling in the missing characters, and then the standardized log structure can be normally parsed. It not only has a high fault tolerance rate, but also avoids the problem of low efficiency caused by only using regular expressions for one-to-one matching of non-standardized log structures in the prior art.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a method for automatically processing a non-standardized log structure provided according to Embodiment 1 of the present invention;

[0025] Figure 2 It is a reference diagram of an abstract key-value pair string applicable to the embodiment of the present invention;

[0026] Figure 3 is a flowchart of another method for automatically processing a non-standardized log structure provided according to Embodiment 2 of the present invention;

[0027] Figure 4 is a schematic diagram of a non-standardized log structure applicable to an embodiment of the present invention;

[0028] Figure 5 is a structural schematic diagram of a non-standardized log structure automatic processing device provided according to Embodiment 3 of the present invention;

[0029] Figure 6 It is a structural schematic diagram of an electronic device for implementing a method for automatically processing a non-standardized log structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a method for automatically processing a non-standardized log structure provided in the first embodiment of the present invention. This embodiment is applicable to the case of automatically processing a non-standardized log structure. The method can be executed by a non-standardized log structure automatic processing device. The non-standardized log structure automatic processing device can be implemented in the form of hardware and / or software. The non-standardized log structure automatic processing device can be configured in a parser that can automatically parse log data. Figure 1 As shown, the method includes:

[0034] S110, identifying a target log to be processed, and when it is determined according to an automatic parsing rule that the target log is a non-standardized log structure, performing feature abstraction and aggregation to obtain an abstract key-value pair string.

[0035] The abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair.

[0036] Since the equipment will generate a huge amount of log information during operation, in order to facilitate accurate analysis of the log, usually, manual labeling is generally adopted to divide the target log to be processed into two parts. The first part is the regular log content, which can be labeled and processed one by one according to the position or characteristics using the regular expression parsing method; the second part is also called the message structure content, which needs to be automatically processed according to the key name: key value rule. However, when the content of the second part is incomplete, that is, when the message structure content becomes non-standardized content, the existing technology cannot perform automatic parsing normally. In the implementation of the present invention, the automatic parsing rule refers to a pre-set judgment condition for determining whether the current target log is a non-standardized log structure. When the automatic parsing rule is met, the subsequent processing can be automatically performed using the method provided in the embodiment of the present invention.

[0037] Therefore, in an embodiment of the present invention, it is first necessary to determine whether the acquired target log to be processed is non-standardized. When the message structure content in the target log to be processed is non-standardized content, an automatic processing method for a non-standardized log structure described in this scheme is executed.

[0038] Feature abstraction and aggregation refer to ignoring the actual content of the message structure in the log to be processed and abstracting it into characters of a unified form in order to facilitate subsequent log processing. For example, the original text is a string of "GlobalSeq": "10A07202512070000121671", "ServiceCode": "F2030001", which becomes "str": "str", "str": "str" ​​after abstraction and aggregation, that is, an abstract key-value pair string.

[0039] Easy to understand, such as Figure 2 As shown, it is a reference diagram of the abstract key-value pair string after abstract aggregation, wherein a key-value pair can be "str": "str", and a structure such as {"str": "str"} containing at least one key-value pair "str": "str" ​​is called an object group.

[0040] S120. Identify target delimiters in the abstract key-value pair string, wherein the target delimiters include: key name-key value delimiters, object group delimiters, key-value pair delimiters, and key name-key value identifiers.

[0041] A key-value pair is an expression that reflects a simple mapping relationship between data, including a key name and a key value, and is often expressed as key:value. Therefore, the key name and key value separator is a separator located between the key name and the key value; the key value pair separator is located in the middle of two key value pairs and is used to separate two key value pairs; the key name and key value identifier is filled with a specific key name string or key value string.

[0042] Since the abstract key-value pair string contains the above-mentioned multiple types of target separators, only by identifying all the separators and classifying them can subsequent automated processing be better performed.

[0043] Further, identifying the key name and key value separators in the abstract key-value pair string may include:

[0044] Identify all separators in the abstract key-value pair string;

[0045] Count the number of each separator respectively, and arrange them in order from most to least to get a separator set;

[0046] Select the separator that ranks first in the separator set as the key name and key value separator.

[0047] The separators in the separator set are sorted from left to right according to the number, and the separator ranked first in number is ranked first in the separator set.

[0048] Optionally, in the abstract key-value pair string, identifying the object group separator may include:

[0049] Traverse all key name and key value separators in the abstract key-value pair string, and locate the first occurrence of the key name and key value separator;

[0050] The object group separator is determined by the position where the key-value separator first appears.

[0051] Specifically, the object group separator appears before the first appearance of the key name and key value separator. Since a group of object group separators includes a left object group separator and a right object group separator, the one that appears before the key name and key value separator specifically refers to the left object group separator, that is, the "{" in an object group of the form {"str":"str"}.

[0052] In addition, since the key name and key value identifier is filled with a specific key name and key value string, that is, in an abstract key value pair string of the form "str": "str", "str": "str", the separator adjacent to the key name and key value separator can be used as the key name and key value identifier, that is, " " in the example.

[0053] Similarly, since the key-value pair separator is located in the middle of two key-value pairs, it is used to separate two key-value pairs. That is, in an abstract key-value pair string of the form "str":"str","str":"str", after the key-value pairs and key-value identifiers are identified according to the key-name key-value separator, the first separator that appears after the last key-name key-value identifier in the current key-value pair can be used as the key-value pair separator, that is, "," in the above example.

[0054] S130. Identify the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and fill in the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure.

[0055] After classifying each separator in the abstract key-value pair string as a certain target separator, first determine the position of the missing characters in the abstract key-value pair string and the type of missing characters, where the missing character type may include the missing specific character content and the missing separator, etc. By comparing with the target log to be processed before feature abstraction and aggregation, the position of the missing character type in the original text of the target log to be processed can be located according to the missing character position, and then the target log to be processed is filled according to the preset character supplementation rules, and the filled non-standardized structure becomes a standardized log structure.

[0056] The technical solution of the embodiment of the present invention identifies the target log to be processed, and when it is determined that the target log is a non-standardized log structure according to the automatic parsing rules, it performs feature abstraction and aggregation to obtain an abstract key-value pair string; and in the abstract key-value pair string, identifies the target separator; identifies the missing character position and the missing character type in the abstract key-value pair string according to the target separator, and fills the missing character position matched in the target log with a supplementary character that matches the missing character type to obtain a standardized log structure. The technical solution of the embodiment of the present invention can automatically determine the non-standardized log structure, and obtain a standardized log structure by identifying missing characters and automatically filling in the missing characters, and then the standardized log structure can be parsed normally, which not only has a high fault tolerance rate, but also avoids the problem of low efficiency caused by the one-to-one matching of non-standardized log structures using regular expressions in the prior art.

[0057] Embodiment 2

[0058] Figure 3 This is a flowchart of another method for automatically processing a non-standardized log structure provided by Embodiment 2 of the present invention. This embodiment is based on the above embodiment and is refined. Figure 3 As shown, the method includes:

[0059] S310: Obtain historical non-standardized log structure data, and read all delimiters in the historical non-standardized log structure data.

[0060] The historical non-standardized log structure is obtained by collecting and storing the non-standardized structure data encountered in the past log information parsing process. Since there are a large number of separator data in the historical non-standardized log structure, the historical non-standardized log structure can also be understood as a sample library of separators.

[0061] S320: construct multiple separator relationship combinations according to all the separators to form automatic parsing rules, wherein the two separators included in each separator relationship combination do not appear adjacent to each other in the standardized log structure.

[0062] By collecting historical non-standardized log structure data, a large number of separator data samples can be obtained, where the separators can be ":", "{", "," and so on. Based on the separators obtained above, a sufficiently rich separator relationship combination can be pre-built, for example, separator relationship combination 1: [: and,]; separator relationship combination 2: [{ and}], etc.

[0063] Correspondingly, the above-mentioned separator relationships can be combined as automatic parsing rules.

[0064] S330, identifying a target log to be processed, and when it is determined according to the automatic parsing rule that the target log is a non-standardized log structure, performing feature abstraction and aggregation to obtain an abstract key-value pair string.

[0065] Optionally, determining that the target log is a non-standardized log structure according to an automatic parsing rule includes:

[0066] Loop through each separator included in the target log, and count the number of hits of the target log to each separator relationship combination;

[0067] If it is determined that the number of hits of the target log to the at least two target separator relationship combinations exceeds a preset number threshold, determining the target log as a candidate log;

[0068] It is detected whether the candidate log includes two adjacent characters that match the target separator relationship combination. If so, the candidate log is determined as a non-standardized log structure.

[0069] Specifically, the automatic parsing rules are determined as follows: As can be seen from S220, each pair of separator relationship combinations can be constructed by any two separators, separator relationship combination 1: [: and,]; separator relationship combination 2: [{ and}], etc. When the number of at least two pairs of separator combinations traversed in the target log to be processed exceeds a preset value, it means that the current target log to be processed contains multiple separator relationship combinations, and then it is determined as a candidate log.

[0070] It is easy to understand that since the two separators contained in the separator relationship combination in the standardized log structure cannot appear adjacent to each other, in a specific example, if "{" and "}" are identified to be adjacent to each other in the candidate log (separator relationship combination 2 in S220), it means that there are missing characters in the candidate log, resulting in the adjacent appearance of the separator relationship combination. At this time, the current candidate log can be determined as a non-standardized log structure. Figure 4 A schematic diagram showing a non-standardized log structure.

[0071] S340. Identify target delimiters in the abstract key-value pair string, wherein the target delimiters include: key name-key value delimiters, object group delimiters, key-value pair delimiters, and key name-key value identifiers.

[0072] S350, identifying the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and filling the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure.

[0073] Further, identifying the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and filling the missing character position matched in the target log with a supplementary character matching the missing character type may include:

[0074] Locate a key-value pair region in the abstract key-value pair string according to the key-value pair separator;

[0075] Identify the missing key-value pair area according to the inclusion of the key name-key value separator and the key name-key value identifier in each key-value pair area;

[0076] According to the arrangement of the key name-key value separators and the key name-key value identifiers in the standard key-value pair area, the missing character position and the missing character type are identified in each missing key-value pair area, and the matching missing character position in the target log is filled with supplementary characters matching the missing character type.

[0077] When the key-value pair separator is determined, the area where the incomplete key-value pair exists can be locked by identifying the key-value pair separator in the abstract key-value pair string, and then the specific missing content can be identified in the locked area. The locked area is the specific missing position of the character in the abstract key-value pair string, and the character missing type can include string missing and separator missing. The corresponding character missing position of the target log to be processed is found through the specific missing position of the character in the abstract key-value pair string, and then the missing supplementary characters are filled in the target log to be processed.

[0078] Specifically, if there is a missing string, you need to first add a pair of key name and key value identifiers at the missing position, and then add a blank string in the key name and key value identifier. Among them, the blank string does not affect the expression of the intention of the field content after parsing. If there is a missing separator, you need to determine the type of the missing separator based on the predetermined target separator, and then supplement it in a targeted manner.

[0079] Optionally, identifying the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and filling the missing character position matched in the target log with a supplementary character matching the missing character type, may also include:

[0080] Construct a separator standard format mapping table based on the JSON format;

[0081] Searching the separator standard format mapping table for a symmetric separator corresponding to the object group separator;

[0082] The number of all object group separators in the abstract key-value pair string is counted, and a symmetric object group separator equal to the number of object group separators is generated.

[0083] In an embodiment of the present invention, the obtained character strings can be uniformly parsed into JSON format, and the writing format of JSON is key name-key value, and each delimiter also has a fixed form requirement, such as: "", {}, [], etc. Therefore, by constructing a delimiter standard format mapping table according to the form requirements of the delimiter in advance, it is possible to find the corresponding single delimiter on the other side by identifying the single delimiter on one side of the delimiter. For example, the single delimiter on the right side of the curly brace "}" is found according to the single delimiter on the left side of the curly brace "{". For ease of understanding, if the single delimiter on the left side of the curly brace "{" is called the object group delimiter, then the single delimiter on the right side of the curly brace "}" is called the symmetric delimiter.

[0084] By traversing the number of object group separators in the target log to be processed, the number of symmetric separators to be supplemented is determined, and a certain number of symmetric separators are supplemented at the end of the target log to be processed.

[0085] The technical solution of the embodiment of the present invention improves the automatic processing method of non-standardized log structures by refining the overall solution. Specifically, it further explains the determination method for non-standardized log structures and the determination method for character missing positions and character missing types. While realizing the recognition and determination of non-standardized log structures, the non-standardized log structures can be accurately filled with characters or separators, avoiding the drawbacks of low efficiency and easy misjudgment caused by relying on manual experience to judge non-standardized logs.

[0086] Embodiment 3

[0087] Figure 5 This is a schematic diagram of a non-standardized log structure automatic processing device provided in Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0088] An abstract key-value pair string determination module 510 is used to identify a target log to be processed, and when the target log is determined to be a non-standardized log structure according to the automatic parsing rule, perform feature abstraction and aggregation to obtain an abstract key-value pair string;

[0089] Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair;

[0090] A target delimiter determination module 520 is used to identify a target delimiter in the abstract key-value pair string, wherein the target delimiter includes: a key name key value delimiter, an object group delimiter, a key value pair delimiter, and a key name key value identifier;

[0091] The standardized log structure generation module 530 is used to identify the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and fill the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure.

[0092] The technical solution of the embodiment of the present invention is that the abstract key-value pair string determination module 510 identifies the target log to be processed, and when it is determined that the target log is a non-standardized log structure according to the automatic parsing rules, it performs feature abstraction and aggregation to obtain an abstract key-value pair string; the target delimiter determination module 520 identifies the target delimiter in the abstract key-value pair string; the standardized log structure generation module 530 identifies the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and fills the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure. The technical solution of the embodiment of the present invention can automatically determine the non-standardized log structure, and obtain a standardized log structure by identifying missing characters and automatically filling in the missing characters, and then the standardized log structure can be normally parsed, which not only has a high fault tolerance rate, but also avoids the problem of low efficiency caused by the one-to-one matching of non-standardized log structures using regular expressions in the prior art.

[0093] Based on the above embodiments, before determining that the target log is a non-standardized log structure according to the automatic parsing rule, the following steps may be included:

[0094] Obtaining historical non-standardized log structure data, and reading all separators in the historical non-standardized log structure data;

[0095] A plurality of separator relationship combinations are constructed according to all the separators to form an automatic parsing rule, wherein the two separators included in each separator relationship combination do not appear adjacently in the standardized log structure.

[0096] Based on the above embodiments, the abstract key-value pair string determination module 510 may include:

[0097] A separator loop traversal unit, used for looping through each separator included in the target log, and counting the number of hits of the target log to each separator relationship combination;

[0098] a candidate log determining unit, configured to determine the target log as a candidate log if it is determined that the number of hits of the target log to at least two target separator relationship combinations exceeds a preset number threshold;

[0099] The non-standardized log structure determination unit is used to detect whether the candidate log includes two adjacent characters that hit the target separator relationship combination, and if so, determine the candidate log as a non-standardized log structure.

[0100] Based on the above embodiments, the target delimiter determination module 520 may include:

[0101] A separator identification unit, used to identify all separators in the abstract key-value pair string;

[0102] The separator number counting unit is used to count the number of each separator respectively, and arrange them in order from most to least to obtain a separator set;

[0103] The key name and key value separator determination unit is used to select the separator that ranks first in the separator set as the key name and key value separator.

[0104] Based on the above embodiments, the target delimiter determination module 520 may further include:

[0105] A name-key-value separator locating unit, used for traversing all the key-name-key-value separators in the abstract key-value pair string, and locating the position where the key-name-key-value separator first appears;

[0106] The object group separator determination unit is used to determine the object group separator according to the position where the key value separator first appears.

[0107] Based on the above embodiments, the standardized log structure generation module 530 may include:

[0108] A key-value pair region positioning unit, used to locate a key-value pair region in the abstract key-value pair string according to the key-value pair separator;

[0109] A missing key-value pair region identifying unit, used to identify the missing key-value pair region according to the inclusion of key name-key value separators and key name-key value identifiers in each key-value pair region;

[0110] The supplementary character filling unit is used to identify the character missing position and the character missing type in each missing key-value pair area according to the arrangement of the key name-key value separators and the key name-key value identifiers in the standard key-value pair area, and fill the matching character missing position in the target log with the supplementary character matching the character missing type.

[0111] Based on the above embodiments, the standardized log structure generating module 530 may further include:

[0112] A separator standard format mapping table construction unit is used to construct a separator standard format mapping table according to the JSON format;

[0113] A symmetric separator searching unit, configured to search the separator standard format mapping table for a symmetric separator corresponding to the object group separator;

[0114] The symmetric object group separator generating unit is used to count the number of all object group separators of the abstract key-value pair string and generate symmetric object group separators equal to the number of object group separators.

[0115] An automatic processing device for a non-standardized log structure provided by an embodiment of the present invention can execute an automatic processing method for a non-standardized log structure provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0116] Embodiment 4

[0117] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a non-standardized log structure automatic processing method.

[0121] That is, identifying the target log to be processed, and when it is determined according to the automatic parsing rule that the target log is a non-standardized log structure, performing feature abstraction and aggregation to obtain an abstract key-value pair string;

[0122] Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair;

[0123] In the abstract key-value pair string, identifying a target separator, wherein the target separator includes: a key name key value separator, an object group separator, a key value pair separator, and a key name key value identifier;

[0124] The character missing position and the character missing type are identified in the abstract key-value pair string according to the target delimiter, and supplementary characters matching the character missing type are filled in the matching character missing position in the target log to obtain a standardized log structure.

[0125] In some embodiments, a non-standardized log structure automatic processing method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the non-standardized log structure automatic processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute a non-standardized log structure automatic processing method in any other appropriate manner (for example, by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically processing a non-standardized log structure, characterized in that: include: Identify the target log to be processed, and when it is determined according to the automatic parsing rule that the target log is a non-standardized log structure, perform feature abstraction and aggregation to obtain an abstract key-value pair string; Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair; In the abstract key-value pair string, identifying a target separator, wherein the target separator includes: a key name key value separator, an object group separator, a key value pair separator, and a key name key value identifier; According to the target delimiter, the missing character position and the missing character type are identified in the abstract key-value pair string, and the missing character position matched in the target log is filled with a supplementary character matching the missing character type to obtain a standardized log structure; wherein the missing character position is a locked area in the abstract key-value pair string where an incomplete key-value pair exists; the missing character type includes the missing specific character content and the missing delimiter; Before determining that the target log is a non-standardized log structure according to the automatic parsing rule, the method further includes: Obtaining historical non-standardized log structure data, and reading all separators in the historical non-standardized log structure data; Constructing a plurality of separator relationship combinations according to all the separators to form an automatic parsing rule, wherein the two separators included in each separator relationship combination do not appear adjacent to each other in the standardized log structure; The step of determining, according to the automatic parsing rule, that the target log is a non-standardized log structure includes: Loop through each separator included in the target log, and count the number of hits of the target log to each separator relationship combination; If it is determined that the number of hits of the target log to the at least two target separator relationship combinations exceeds a preset number threshold, determining the target log as a candidate log; It is detected whether the candidate log includes two adjacent characters that match the target separator relationship combination. If so, the candidate log is determined as a non-standardized log structure.

2. The method according to claim 1, characterized in that Identify key name and key value separators in the abstract key-value pair string, including: Identify all separators in the abstract key-value pair string; Count the number of each separator respectively, and arrange them in order from most to least to get a separator set; Select the separator that ranks first in the separator set as the key name and key value separator.

3. The method according to claim 1, characterized in that In the abstract key-value pair string, identify the object group separator, including: Traverse all key name and key value separators in the abstract key-value pair string, and locate the first occurrence of the key name and key value separator; The object group separator is determined by the position where the key-value separator first appears.

4. The method according to any one of claims 1 to 3, characterized in that: Identifying a missing character position and a missing character type in the abstract key-value pair string according to the target delimiter, and filling in a supplementary character matching the missing character type at the matching missing character position in the target log, including: Locate a key-value pair region in the abstract key-value pair string according to the key-value pair separator; Identify the missing key-value pair area according to the inclusion of the key name-key value separator and the key name-key value identifier in each key-value pair area; According to the arrangement of the key name-key value separators and the key name-key value identifiers in the standard key-value pair area, the missing character position and the missing character type are identified in each missing key-value pair area, and the matching missing character position in the target log is filled with supplementary characters matching the missing character type.

5. The method according to any one of claims 1 to 3, characterized in that: The invention also includes: identifying the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and filling the missing character position matched in the target log with a supplementary character matching the missing character type; Construct a separator standard format mapping table based on the JSON format; Searching the separator standard format mapping table for a symmetric separator corresponding to the object group separator; The number of all object group separators in the abstract key-value pair string is counted, and a symmetric object group separator equal to the number of object group separators is generated.

6. A non-standardized log structure automatic processing device, characterized in that: include: An abstract key-value pair string determination module is used to identify a target log to be processed, and when the target log is determined to be a non-standardized log structure according to an automatic parsing rule, perform feature abstraction and aggregation to obtain an abstract key-value pair string; Wherein, the abstract key-value pair string includes at least one key-value pair and at least one object group consisting of the at least one key-value pair; A target delimiter determination module is used to identify a target delimiter in the abstract key-value pair string, wherein the target delimiter includes: a key name key value delimiter, an object group delimiter, a key value pair delimiter and a key name key value identifier; A standardized log structure generation module is used to identify the missing character position and the missing character type in the abstract key-value pair string according to the target delimiter, and fill the missing character position matched in the target log with a supplementary character matching the missing character type to obtain a standardized log structure; wherein the missing character position is a locked area in the abstract key-value pair string where an incomplete key-value pair exists; the missing character type includes the missing specific character content and the missing delimiter; Before determining that the target log is a non-standardized log structure according to the automatic parsing rule, the method further includes: Obtaining historical non-standardized log structure data, and reading all separators in the historical non-standardized log structure data; Constructing a plurality of separator relationship combinations according to all the separators to form an automatic parsing rule, wherein the two separators included in each separator relationship combination do not appear adjacent to each other in the standardized log structure; The abstract key-value pair string determination module includes: A separator loop traversal unit, used for looping through each separator included in the target log, and counting the number of hits of the target log to each separator relationship combination; a candidate log determining unit, configured to determine the target log as a candidate log if it is determined that the number of hits of the target log to at least two target separator relationship combinations exceeds a preset number threshold; The non-standardized log structure determination unit is used to detect whether the candidate log includes two adjacent characters that hit the target separator relationship combination, and if so, determine the candidate log as a non-standardized log structure.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for automatically processing a non-standardized log structure according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for automatically processing a non-standardized log structure according to any one of claims 1 to 5 when executed.

Images