Method, apparatus, device and program product for log preprocessing

Abstract

One or more embodiments of the present specification relate to a method, an apparatus, a computing device, a computer-readable storage medium, and a computer program product for log preprocessing. The method includes determining a plurality of fields in a log to be processed. The method also includes, for each field of the plurality of fields, determining a first encoding associated with characters of the field and a second encoding associated with a length of the field. Further, the method also includes determining a target encoding of the field based on the first encoding and the second encoding.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification generally relate to the field of computer technology, and more particularly, to a method, an apparatus, a computing device, a computer readable storage medium, and a computer program product for log preprocessing. BACKGROUND

[0002] A log of a program running is a record information generated by the program during execution, which contains various states, events and errors of the program running. The log information usually includes timestamp, log level (such as INFO, WARN, ERROR, etc.), log content, etc. These log information is very important to developers and operation and maintenance personnel, because they can help them understand the running situation of the program, and timely find and solve problems. These log information can also be used to train machine learning models for the development of log diagnosis systems.

[0003] Log specification refers to the unified standard formulated for the format, content, storage, etc. of the program running log. The purpose of formulating log specification is to improve the readability, maintainability and analyzability of the log, so as to reduce the difficulty and cost of log management. The log specification can include unified format and separator, etc. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide a log preprocessing method, apparatus, computing device, computer readable storage medium and computer program product, which can use statistical thinking instead of traditional logical thinking, while only retaining the length characteristics of the fields in disordered logs and the most obvious keywords therein, to normalize disordered logs without relying on human resources to write a large number of codes to adapt to each log format.

[0005] In a first aspect of the present specification, a method for log preprocessing is provided. The method includes determining a plurality of fields in a log to be processed. The method also includes, for each field of the plurality of fields, determining a first encoding associated with characters of the field and a second encoding associated with a length of the field. In addition, the method further includes determining a target encoding of the field based on the first encoding and the second encoding.

[0006] In a second aspect of the present specification, an apparatus for log preprocessing is provided. The apparatus includes a field determination unit configured to determine a plurality of fields in a log to be processed. The apparatus also includes a fuzzy encoding unit configured to, for each field of the plurality of fields, determine a first encoding associated with characters of the field and a second encoding associated with a length of the field. In addition, the apparatus further includes a target encoding determination unit configured to determine a target encoding of the field based on the first encoding and the second encoding.

[0007] In a third aspect of the present specification, there is provided a computing device comprising: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to perform the method as claimed in the first aspect of the present specification.

[0008] In a fourth aspect of the present specification, there is provided a non-transitory computer storage medium comprising machine executable instructions that, when executed by a device, cause the device to perform the method as claimed in the first aspect of the present specification.

[0009] In a fifth aspect of the present specification, there is provided a computer program product comprising machine executable instructions that, when executed by a device, cause the device to perform the method as claimed in the first aspect of the present specification.

[0010] It is to be understood that the description of the background of the application in this summary section is not necessarily intended to outline key or essential features of the embodiments of the present specification nor does it necessarily describe essential features of the present disclosure. Other features, aspects, and advantages of the present specification will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other objects, features and advantages of embodiments of the present specification will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, various embodiments of the present specification are illustrated by way of example and not limitation in which:

[0012] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present specification can be implemented is shown;

[0013] Figure 2 A flow diagram illustrating a method for log pre-processing according to some embodiments of the present specification is shown;

[0014] Figure 3 A structural schematic diagram of a log aggregation encoder according to some embodiments of the present specification is shown;

[0015] Figures 4A-4D A schematic diagram illustrating the conversion of a field to a target encoding according to some embodiments of the present specification is shown;

[0016] Figure 5 A block diagram of an apparatus for log pre-processing according to some embodiments of the present specification is shown; and

[0017] Figure 6 A block diagram of an electronic device according to some embodiments of the present specification is shown.

[0018] In all the drawings, like or similar reference numerals refer to like or similar elements. DETAILED DESCRIPTION

[0019] Embodiments of the present specification will be described in more detail by referring to the drawings. While certain embodiments of the present specification are shown in the drawings, it is understood that the present specification can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather should be construed to encompass all modifications equivalent in meaning and scope to the present specification. It is understood that the drawings and embodiments of the present specification are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of embodiments of the present specification, the term "include" and its conjugates are to be understood as open-ended, i.e., "including but not limited to". The term "based on" is to be understood as "based, at least in part, on". The term "one embodiment" or "an embodiment" means "at least one embodiment". The terms "a plurality" and "a plurality of" mean "two or more". The terms "first", "second", and the like, can refer to different or identical objects. Other explicitly and implicitly recited definitions can also be found below.

[0021] As described above, the log of program running and its specification play an important role in the process of software development and operation. However, when a large team uses a log uniformly, there are often problems such as no unified specification of the log and no unified printing habits, which greatly hinders the training of the model, thereby affecting the development of the log diagnosis system. Under the traditional method, when encountering the situation that the format in the log is not unified, a large amount of code needs to be written to adapt to each log format until they are unified, which is very time-consuming and laborious, and needs manpower to follow up and maintain, and is not universal.

[0022] Therefore, in the embodiments of the present specification, a method for log preprocessing is provided. First, a plurality of fields are obtained from a log to be processed. Then, for each field, a first code is obtained based on character information of the field, and a second code is obtained based on length information of the field, and finally a unified target code is obtained. In this way, the disordered log can be standardized while only retaining the length characteristics of the fields in the disordered log and the most obvious keywords therein. Therefore, the embodiments of the present specification can save human resources, improve efficiency, and have universality.

[0023] Figure 1 A schematic diagram of an environment 100 in which embodiments of the present specification can be implemented is shown. As Figure 1As shown, environment 100 may include an unordered log 102, a log aggregation encoder 104, a log keyword library 106, and normalization encoding 108. The unordered log 102 may be a log file lacking a unified format, with an unclear structure, and difficult to understand and trace. The unordered log 102 may be generated by conflicts arising from simultaneous editing by multiple people, a lack of clear recording standards, technical limitations, or human error. In some embodiments, the unordered log 102 may be stored as a text file on a local device and can be read directly from the local device. Furthermore, the unordered log 102 may also originate from storage devices such as databases or cloud storage.

[0024] like Figure 1 As shown, the unordered log 102 can be provided to the log aggregation encoder 104. The log aggregation encoder 104 can be deployed on a standalone physical network server, or on a network server cluster or distributed system composed of multiple physical network servers. It can also be deployed on a cloud network server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, security services, and big data and artificial intelligence platforms. A structural diagram of the log aggregation encoder 104 will be shown below. Figure 3 Detailed explanation will follow.

[0025] like Figure 1 As shown, the log aggregation encoder 104 can refer to the log keyword library 106 to transform the unordered log 102 into normalized encoding 108. The log keyword library 106 is a predefined thesaurus used to match meaningful strings from the log. The log keyword library 106 can include various keywords, phrases, or patterns related to the log content to help identify, extract, and analyze useful information in the log.

[0026] In some embodiments, the log keyword library 106 can be manually constructed by collecting and organizing keywords and phrases according to actual needs and log characteristics. In some embodiments, natural language processing (NLP) or machine learning techniques can also be used to automatically extract keywords and phrases from a large amount of log data to construct the log keyword library 106. Furthermore, a combination of manual construction and automatic extraction methods can be used to build the log keyword library 106. One or more embodiments of this specification do not limit the source of the log keyword library 106.

[0027] Normalized encoding 108 is an important form of ensuring log data consistency, readability, and maintainability. Normalized encoding 108 has a consistent format, delimiters, etc. Furthermore, normalized encoding 108 can achieve a higher compression ratio compared to the original log file while preserving as much valid log information as possible, thus saving storage resources. In some embodiments, normalized encoding 108 can be used to train machine learning models for the development of log diagnostic systems.

[0028] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this specification. Embodiments of this specification can also be applied to other environments with different structures and / or functionalities.

[0029] The following will combine Figure 2 to Figure 4D The process according to embodiments of this specification is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It should be understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0030] Figure 2 A schematic flowchart of a method 200 for log preprocessing according to some embodiments of this specification is shown. In some embodiments, method 200 may be, for example, Figure 1 The log aggregation encoder 104 shown is used for implementation. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0031] like Figure 2 As shown in box 210, method 200 may include: identifying multiple fields in a log to be processed. Logs are a critical component of system administration and troubleshooting. A complete log entry typically contains multiple fields to provide sufficient information for understanding and analyzing the problem. In different types of logs, different fields are usually separated by delimiters so that tools such as log parsers can accurately identify each field and extract its value.

[0032] In some embodiments, the log aggregation encoder 104 may first determine the type of delimiter used in the log to be processed (e.g., space, colon, comma, etc.). Then, based on the delimiter type, the log aggregation encoder 104 may determine multiple delimiters in the log to be processed. Based on these delimiters, the log aggregation encoder 104 can easily determine multiple fields in the log to be processed between pairs of delimiters.

[0033] In some cases, unordered logs lack a unified delimiter, making it impossible to extract fields for standardized encoding. In such situations, the log aggregation encoder 104 can rely on a predefined keyword library to segment the fields. This keyword library includes various keywords, phrases, or patterns related to the log content, enabling the matching of words or strings with actual value from the log.

[0034] In some embodiments, the log aggregation encoder 104 can first match a plurality of keywords from the log to be processed based on a keyword library. Then, the log aggregation encoder 104 can determine a field by concatenating a current keyword and a string between the current keyword and a next keyword. If a log line starts with a meaningless string, the meaningless string can be replaced with a custom identifier, e.g., “nonsense”. In this way, the log aggregation encoder 104 is able to self- separate a plurality of fields in a disordered log without uniform delimiters for subsequent normalization encoding operations.

[0035] At block 220, the method 200 can include determining, for each field of the plurality of fields, a first encoding associated with characters of the field and a second encoding associated with a length of the field. The obfuscation encoding is an encoding manner for reducing the amount of text information while preserving enough key features for identification, which can be used to obtain the first encoding and the second encoding associated with the log field.

[0036] In some embodiments, the log aggregation encoder 104 can determine one or more keywords from the field, where the one or more keywords have actual valuable information. Then, the log aggregation encoder 104 can obtain the first encoding by concatenating the one or more keywords. In this way, the first encoding is able to preserve key character information in the field.

[0037] In some embodiments, the log aggregation encoder 104 can obtain a candidate word set from the field based on a keyword library. Then, the log aggregation encoder 104 can determine a candidate word in the candidate word set with a length greater than a threshold as a keyword. In this way, the amount of text information can be further reduced while preserving enough key features for identification.

[0038] In some embodiments, the log aggregation encoder 104 can determine a division granularity associated with a number of bits of the length of the field based on the length of the field. Then, the log aggregation encoder 104 can determine the second encoding based on the division granularity. In this way, the length feature of the field can be preserved while reducing the amount of text information for inspection and diagnosis of the log.

[0039] In some embodiments, the log aggregation encoder 104 can determine a divisor of the length of the field based on the number of bits of the length of the field, and then perform an integer division operation on the length of the field based on the divisor. The log aggregation encoder 104 can then determine the result of the integer division operation as the second encoding. For example, for a length of 23, the log aggregation encoder 104 can first determine that the division granularity is two bits. Then, the log aggregation encoder 104 can determine that the divisor is 10 to obtain an integer division result of “2” as the second encoding. For another example, for a length of 347, the log aggregation encoder 104 can first determine that the division granularity is three bits. Then, the log aggregation encoder 104 can determine that the divisor is 100 to obtain an integer division result of “3” as the second encoding.

[0040] At block 230, the method 200 can include determining a target encoding of the field based on the first encoding and the second encoding. The first encoding includes a feature associated with the characters of the field, and the second encoding includes a feature associated with the length of the field. Thus, by combining the first encoding and the second encoding, the features associated with the characters and the length of the field can be included simultaneously.

[0041] In some embodiments, the log aggregation encoder 104 can determine the first encoding as a prefix of the target encoding, determine the second encoding as a suffix of the target encoding, and then obtain the target encoding of the field by concatenating the prefix and the suffix. Finally, the log aggregation encoder 104 can combine the target encodings of each field in the log to completely transform the log into a canonical format. In this way, the log aggregation encoder 104 can complete the normalization of the unordered log to facilitate the diagnosis and analysis of the log. In some embodiments, the target encoding can also be used to train a machine learning model to develop a log diagnosis system.

[0042] Figure 3 A structural diagram of a log aggregation encoder 300 according to some embodiments of the present specification is shown. The log aggregation encoder 300 can be regarded as a specific implementation of the log aggregation encoder 104 as shown in Figure 1 As shown in Figure 3 The log aggregation encoder 300 can first obtain an unordered log 302. The unordered log 302 can be a log file that lacks a unified format, is unclear in structure, is difficult to understand and trace, and is generated by multiple people editing at the same time, conflicts, lack of clear recording specifications, technical limitations, or human errors, etc. Optionally, the unordered log 302 can be read directly from a local device, or read from a storage device (e.g., a database, cloud storage, etc.).

[0043] Then, at step 304, the log aggregation encoder 300 can determine the separator type (e.g., space, colon, comma, etc.) in the unordered log 302. Based on the separator type, the log aggregation encoder 300 can determine a plurality of separators in the unordered log 302, and thus determine a plurality of fields, including field 306-1, field 306-2, …, field 306-N (collectively referred to as fields 306).

[0044] Then, the log aggregation encoder 300 can obtain a normalized target encoding for each field while preserving the key features of the field. As shown in FIG. 3, taking field 306-3 as an example, the log aggregation encoder 300 can obtain, at step 308, a first encoding related to the characters of field 306-3, and obtain, at step 310, a second encoding related to the length of field 306-3, respectively. Figure 3

[0045] Specifically, at step 308-1, the log aggregation encoder 300 can match a plurality of candidate words in field 306-3 to obtain a candidate word set based on a log keyword library. The log keyword library is a predefined word library containing a large number of meaningful strings. Then, at step 308-2, the log aggregation encoder 300 can further filter the candidate word set to obtain a keyword. For example, the candidate words in the candidate word set with a character length greater than 5 can be determined as keywords. At step 308-3, the log aggregation encoder 300 can concatenate a plurality of keywords to obtain the first encoding related to the characters of field 306-3. If there is no keyword meeting the condition as the first encoding, optionally, a custom keyword can be used to represent that there is no keyword meeting the condition in the field, for example, “nonsense”.

[0046] On the other hand, at step 310-1, the log aggregation encoder 300 can first obtain the length of field 306-3. Then, at step 310-2, the log aggregation encoder 300 can determine a division granularity based on the obtained length. For example, if the length of field 306-3 is 24, the division granularity can be determined as 10. For another example, if the length of field 306-3 is 347, the division granularity can be determined as 100. At step 310-3, the log aggregation encoder 300 can obtain the second encoding related to the length of field 306-3 based on the division granularity. For example, if the length of field 306-3 is 24, the second encoding can be determined as the result of 24 divided by 10, i.e., “2”.

[0047] ​After obtaining the first encoding related to the characters of the field and the second encoding related to the length of the field, the log aggregation encoder 300 can prefix the first encoding as the prefix of the target encoding and suffix the second encoding as the suffix of the target encoding, so as to splice the first encoding and the second encoding to obtain the target encoding for the field. Finally, the log aggregation encoder 300 can combine the target encodings of each field in the unordered log 302 to complete the normalization processing of the unordered log 302. In this way, the log format can be normalized while retaining as much log effective information as possible, and the storage occupation can be reduced.

[0048] Figures 4A-4D A schematic diagram of converting a field into a target encoding is shown according to some embodiments of the present specification. It can be understood that in one or more embodiments of the specification, the meaningless character not present in the log keyword library 106 is represented by “*”, and the length threshold of the preset keyword is 5.

[0049] As Figure 4A shown, the field 401-1 contains the candidate word “inferenceArray” present in the log keyword library 106 and the meaningless character “*”. Since the length of the candidate word “inferenceArray” is 14, which exceeds the preset threshold 5, the log aggregation encoder 104 can determine the candidate word “inferenceArray” as the keyword of the field 401-1. Since the keyword “inferenceArray” is the only keyword in the field 401-1, the log aggregation encoder 104 can determine the first encoding of the field 401-1 as “inferenceArray”.

[0050] In addition, since the total length of the field 401-1 is 23, the log aggregation encoder 104 can determine that the granularity division of the field 401-1 is two digits. Therefore, the log aggregation encoder 104 can divide the total length of the field 401-1 by the smallest number 10 of two digits, and take the obtained result “2” as the second encoding. Then, the log aggregation encoder 104 can splice the obtained first encoding and the second encoding to determine the target encoding 401-2 as “inferenceArray2”.

[0051] Thus, the target encoding 401-2 can normalize the unordered field 401-1 while retaining the length characteristics of the unordered field 401-1 and the most obvious keywords therein, and has a large information compression ratio, saving storage resources.

[0052] As Figure 4BAs shown, field 402-1 contains candidate words "inferenceArray" and "unit". However, since the length of the candidate word "unit" is 4, it does not meet the preset length threshold. Therefore, field 402-1 still only contains the unique keyword "inferenceArray". The log aggregation encoder 104 can determine that the first code of field 402-1 is "inferenceArray". In addition, since the total length of field 402-1 is 33, based on a similar method, the log aggregation encoder 104 can determine that the second code of field 402-1 is "3". Then, the log aggregation encoder 104 can concatenate the first and second codes of field 402-1 to determine the target code 402-2 as "inferenceArray3".

[0053] like Figure 4C As shown, field 403-1 contains the candidate words "inferenceArray" and "oxford," both of which meet the preset length threshold. Therefore, field 402-1 contains two keywords, "inferenceArray" and "oxford." The log aggregation encoder 104 can then concatenate these two keywords, resulting in "inferenceArrayoxford," which is determined as the first code for field 402-1. Furthermore, since the total length of field 403-1 is 33, the log aggregation encoder 104 can determine the second code for field 403-1 as "3" using a similar method. Then, the log aggregation encoder 104 can concatenate the first and second codes of field 403-1 to determine the target code 403-2 as "inferenceArrayoxford3."

[0054] like Figure 4D As shown, field 404-1 contains the candidate word "unit," but since the length of the candidate word "unit" is 4, it does not meet the preset length threshold. Therefore, field 404-1 does not contain any keywords that meet the conditions. In this case, the log aggregation encoder 104 can determine that the first encoding of field 404-1 is "nonsense." Furthermore, since the total length of field 404-1 is 22, based on a similar method described above, the log aggregation encoder 104 can determine that the second encoding of field 404-1 is "2." Then, the log aggregation encoder 104 can concatenate the first and second encodings of field 404-1 to determine the target encoding 404-2 as "nonsense2."

[0055] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and computing device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0056] Figure 5 A schematic block diagram of an apparatus 500 for log preprocessing according to some embodiments of this specification is shown. Figure 5 As shown, the device 500 includes: a field determination unit 502, a fuzzy coding unit 504, and a target coding determination unit 506.

[0057] In some embodiments, the field determination unit 502 is configured to determine multiple fields in the log to be processed; the fuzzy encoding unit 504 is configured to determine a first encoding associated with the characters of the field and a second encoding associated with the length of the field for each of the multiple fields; and the target encoding determination unit 506 is configured to determine a target encoding of the field based on the first encoding and the second encoding.

[0058] In some embodiments, the field determination unit 502 is further configured to determine the delimiter type used in the log to be processed; determine multiple delimiters in the log to be processed based on the delimiter type; and determine multiple fields in the log to be processed based on the multiple delimiters.

[0059] In some embodiments, the fuzzy coding unit 504 is further configured to determine one or more keywords from the field, the one or more keywords having actual valuable information; and to obtain a first code by concatenating the one or more keywords.

[0060] In some embodiments, the fuzzy coding unit 504 is further configured to obtain a set of candidate words from the field based on a keyword library; and to determine candidate words in the set whose length is greater than a threshold as keywords.

[0061] In some embodiments, the fuzzy coding unit 504 is further configured to determine a partitioning granularity associated with the number of bits in the field length based on the field length; and to determine a second code based on the partitioning granularity.

[0062] In some embodiments, the fuzzy coding unit 504 is further configured to determine the divisor of the field length based on the number of bits in the field length; perform an integer division operation on the field length based on the divisor; and determine the result of the integer division operation as a second code.

[0063] In some embodiments, the target encoding determining unit 506 is further configured to determine the first encoding as a prefix of the target encoding; determine the second encoding as a suffix of the target encoding; and obtain the target encoding of the field by concatenating the prefix and the suffix.

[0064] In some embodiments, the field determining unit 502 is further configured to match a plurality of keywords from the log to be processed based on a keyword library; and determine a field by concatenating a current keyword and a string between the current keyword and a next keyword.

[0065] In some embodiments, the target encoding is used for training a machine learning model.

[0066] It should be noted that more actions or steps shown in Figures 1 to 4D may be implemented by the apparatus 500 shown in Figure 5 For example, the apparatus 500 can include more modules or units to implement the actions or steps described above, or Figure 5 Some units or modules shown in may be further configured to implement the actions or steps described above. Here, no further elaboration is repeated.

[0067] Figure 6 A schematic block diagram of an example device 600 that can be used to implement some embodiments of the present specification is shown. As Figure 6 shown, the device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 606 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0068] A plurality of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0069] The computing unit 601 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs, graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the method 200. For example, in some embodiments, the method 200 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the method 200 described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method 200 by any other suitable means, such as by means of firmware.

[0070] In some embodiments, the methods and processes described above can be implemented as computer program products. Computer program products can include computer readable storage media having computer readable program instructions thereon for performing various aspects of the present disclosure.

[0071] Computer readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, semiconductor system, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magnetically encoded device such as magnetic strip cards, an optically encoded device such as a compact disc (CD) or DVD, and / or any suitable combination of the foregoing. Computer readable program instructions can be downloaded to or from computer readable storage media 608. The computer readable program instructions can be downloaded to the device 600 from the computer readable storage media 608 and / or the device 600 can read the computer readable program instructions from the computer readable storage media 608 while the device 600 is in operation. The computer readable program instructions can be downloaded to the device 600 from the computer readable storage media 608 and / or the device 600 can read the computer readable program instructions from the computer readable storage media 608 while the device 600 is in operation. The computer readable program instructions can also be downloaded to the device 600 from the computer readable storage media 608 and / or the device 600 can read the computer readable program instructions from the computer readable storage media 608 while the device 600 is in operation.

[0072] The computer software components described herein can be downloaded from a network, such as the Internet, or another computer network. The computer software components described herein can also be stored on a computer readable storage medium that can be read by a computer system. Examples of computer readable storage mediums include magnetic cassettes, DVDs, CD-ROMs, memory cards, flash drives, and optical data storage devices. The computer software components described herein can also be stored on a computer system that is connected to a network, such as the Internet, or another computer network.

[0073] Computer readable program instructions for carrying out operations of the embodiments of the present specification can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Computer readable program instructions can be executed completely on a user computer, partially on a user computer, as a stand-alone software package, partially on a user computer and partially on a remote computer, or completely on a remote computer or server. In cases involving remote computers, remote computers can be connected to a user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet by using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGA) or programmable logic arrays (PLA), which can execute computer readable program instructions, are personalized by using state information of computer readable program instructions, thereby implementing various aspects of the present specification.

[0074] These computer readable program instructions can be provided to a processing unit of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, thereby producing a machine, so that these instructions, when executed by the processing unit of the computer or other programmable data processing apparatus, produce an apparatus that implements the functions / acts specified in one or more of the blocks in the flowchart and / or block diagram. These computer readable program instructions can also be stored in a computer readable storage medium, which causes the computer, programmable data processing apparatus, and / or other device to work in a specific manner, so that the computer readable medium storing the instructions includes a manufacture that implements the aspects of the functions / acts specified in one or more of the blocks in the flowchart and / or block diagram.

[0075] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0076] The above-described embodiments of the application have been described in connection with what are presently considered to be the most practical and preferred embodiments, and the control system described herein is therefore not limited to the described embodiments but rather can be carried out in other ways than those specifically set forth herein without departing from the spirit and the scope of the application. Numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

[0077] The above-described embodiments of the application have been described in connection with what are presently considered to be the most practical and preferred embodiments, and the control system described herein is therefore not limited to the described embodiments but rather can be carried out in other ways than those specifically set forth herein without departing from the spirit and the scope of the application. Numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

[0078] The above-described embodiments of the application have been described in connection with what are presently considered to be the most practical and preferred embodiments, and the control system described herein is therefore not limited to the described embodiments but rather can be carried out in other ways than those specifically set forth herein without departing from the spirit and the scope of the application. Numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for log preprocessing, comprising: Identify multiple fields in the log to be processed; For each of the plurality of fields, a first code associated with a character of the field and a second code associated with the length of the field are determined; as well as Based on the first encoding and the second encoding, the target encoding of the field is determined.

2. The method according to claim 1, wherein determining the plurality of fields in the log to be processed includes: Determine the delimiter type used in the log to be processed; Based on the delimiter type, determine multiple delimiters in the log to be processed; as well as Based on the multiple delimiters, the multiple fields in the log to be processed are determined.

3. The method of claim 1, wherein determining the first encoding associated with a character of the field comprises: One or more keywords are identified from the fields, and the one or more keywords have actual valuable information; as well as The first code is obtained by concatenating one or more keywords.

4. The method of claim 3, wherein determining one or more first strings from the field comprises: Based on the keyword library, a set of candidate words is obtained from the fields; as well as Candidate words with a length greater than a threshold in the candidate word set are identified as keywords.

5. The method of claim 1, wherein determining the second encoding associated with the length of the field comprises: Based on the length of the field, determine the partitioning granularity associated with the number of bits in the field. as well as The second code is determined based on the granularity of the division.

6. The method of claim 5, wherein determining the second code comprises: Determine the divisor of the field length based on the number of digits in the field. Based on the divisor, perform an integer division operation on the length of the field; as well as The result of the integer division operation is determined as the second code.

7. The method according to claim 1, wherein obtaining the target encoding of the field based on the first encoding and the second encoding comprises: The first code is determined as the prefix of the target code; The second code is determined as the suffix of the target code; as well as The target encoding of the field is obtained by concatenating the prefix and the suffix.

8. The method of claim 1, wherein determining the plurality of fields in the log to be processed includes: Based on the keyword library, multiple keywords are matched from the logs to be processed; as well as A field is determined by concatenating the current keyword and the string between the current keyword and the next keyword.

9. The method of claim 1, wherein the target encoding is used to train a machine learning model.

10. A system for log preprocessing, comprising: The field determination unit is configured to determine multiple fields in the log to be processed. A fuzzy encoding unit is configured to, for each of the plurality of fields, determine a first encoding associated with a character of the field and a second encoding associated with the length of the field; as well as The target encoding determination unit is configured to determine the target encoding of the field based on the first encoding and the second encoding.

11. A computing device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to perform the method as described in any one of claims 1 to 9.

12. A computer storage medium comprising machine-executable instructions that, when executed by a device, cause the device to perform the method as claimed in any one of claims 1 to 9.

13. A computer program product comprising machine-executable instructions that, when executed by a device, cause the device to perform the method as described in any one of claims 1 to 9.

Images