Method, device, equipment and program product for log preprocessing

Abstract

One or more embodiments of the invention relate to a method and apparatus for log preprocessing, a computing device, a computer readable storage medium and a computer program product. The method comprises the steps of determining a plurality of fields in a log to be processed; the method further includes determining, for each field of the plurality of fields, a first code associated with a character of the field and a second code associated with a length of the field. Further, the method includes determining a target code for the field based on the first code and the second code.

Description

Technical Field

[0001] One or more embodiments of the present specification generally relate to the field of computer technology, and more specifically, to a method, apparatus, computing device, computer-readable storage medium, and computer program product for log preprocessing. Background Art

[0002] The program running log is the recorded information generated during the execution of the program, which contains various states, events and errors of the program running. Log information usually includes timestamp, log level (such as INFO, WARN, ERROR, etc.), log content, etc. These log information are very important for developers and operation and maintenance personnel because they can help them understand the operation of the program and find and solve problems in time. These log information can also be used to train machine learning models for the development of log diagnosis systems.

[0003] Log specifications refer to unified standards for the format, content, storage, etc. of program operation logs. The purpose of developing log specifications is to improve the readability, maintainability, and analyzability of logs, thereby reducing the difficulty and cost of log management. Log specifications can include unified formats and separators, etc. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a log preprocessing method, apparatus, computing device, computer-readable storage medium, and computer program product, which can use statistical thinking instead of traditional logical thinking to normalize unordered logs while retaining only the length characteristics of the fields in the unordered logs and the most obvious keywords therein, without relying on human resources to write a large amount of code to adapt to each log format.

[0005] In a first aspect of the present specification, a method for log preprocessing is provided. The method includes determining a plurality of fields in a log to be processed. The method also includes determining, for each of the plurality of fields, a first encoding associated with a character of the field and a second encoding associated with a length of the field. In addition, the method also includes determining a target encoding of the field based on the first encoding and the second encoding.

[0006] In a second aspect of the present specification, a device for log preprocessing is provided. The device includes a field determination unit configured to determine multiple fields in a log to be processed. The device also includes a fuzzy coding unit configured to determine, for each of the multiple fields, a first code associated with a character of the field and a second code associated with a length of the field. In addition, the device also includes a target code determination unit configured to determine a target code of the field based on the first code and the second code.

[0007] In a third aspect of the present specification, a computing device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to execute the method as described in the first aspect of the present specification.

[0008] In a fourth aspect of the present specification, a non-transitory computer storage medium is provided, comprising machine executable instructions, which, when executed by a device, cause the device to perform the method as described in the first aspect of the present specification.

[0009] In a fifth aspect of the present specification, a computer program product is provided, comprising machine executable instructions, which, when executed by a device, cause the device to perform the method as described in the first aspect of the present specification.

[0010] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of this specification, nor are they intended to limit the scope of this disclosure. Other features of this specification will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the various embodiments of the present specification will become more easily understood through the following detailed description with reference to the accompanying drawings. In the accompanying drawings, various embodiments of the present specification will be described in an exemplary and non-limiting manner, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which some embodiments according to the present specification may be implemented;

[0013] Figure 2 A flowchart of a method for log preprocessing according to some embodiments of the present specification is shown;

[0014] Figure 3 A schematic diagram of the structure of a log aggregation encoder according to some embodiments of the present specification is shown;

[0015] Figure 4A-4D A schematic diagram showing converting a field into a target code according to some embodiments of the present specification is shown;

[0016] Figure 5 A block diagram showing an apparatus for log preprocessing according to some embodiments of the present specification; and

[0017] Figure 6 A block diagram of an electronic device according to some embodiments of the present specification is shown.

[0018] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0019] The embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present specification are shown in the accompanying drawings, it should be understood that the present specification can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of this specification, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] As mentioned above, program running logs and their specifications play an important role in software development and operation and maintenance. However, when a large team uses a unified log, they often encounter problems such as no unified log specifications and no unified printing habits. This has brought great obstacles to model training and thus affected the development of log diagnostic systems. Under the traditional method, when encountering inconsistent formats in logs, a large amount of code will be written to adapt to each log format until they are unified, which is very time-consuming and labor-intensive, and requires continuous follow-up maintenance by manpower, and is not universal.

[0022] To this end, a method for log preprocessing is provided in an embodiment of the present specification. First, multiple fields are obtained from the log to be processed. Then, for each field, a first code is obtained based on the character information of the field, and a second code is obtained based on the length information of the field, and finally a unified target code is obtained. In this way, the disordered log can be standardized while retaining only the length characteristics of the fields in the disordered log and the most obvious keywords therein. Therefore, the embodiments of the present specification can save human resources, improve efficiency, and have universal applicability.

[0023] Figure 1 1 shows a schematic diagram of an environment 100 in which various embodiments of the present specification can be implemented. Figure 1As shown, the environment 100 may include an unordered log 102, a log aggregation encoder 104, a log keyword library 106, and a standardized code 108. The unordered log 102 may be a log file that lacks a unified format, has an unclear structure, is difficult to understand and track. The unordered log 102 may be generated by conflicts generated by multiple people editing at the same time, lack of clear recording specifications, technical limitations, or human errors. In some embodiments, the unordered log 102 may be stored in a local device in the form of a text file and can be read directly from the local device. In addition, the unordered log 102 may also come from some storage devices, such as databases, cloud storage, etc.

[0024] like Figure 1 As shown, the unordered log 102 can be provided to the log aggregation encoder 104. The log aggregation encoder 104 can be deployed on an independent physical network server, or on a network server cluster or distributed system consisting of multiple physical network servers, or on a cloud network server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, security services, and big data and artificial intelligence platforms. The structural diagram of the log aggregation encoder 104 will be described in the following. Figure 3 Detailed description is given in .

[0025] like Figure 1 As shown, the log aggregation encoder 104 can refer to the log keyword library 106 to convert the unordered log 102 into a standardized code 108. The log keyword library 106 is a predefined word library for matching meaningful strings from the log. The log keyword library 106 can include various keywords, phrases or patterns related to the log content to help identify, extract and analyze useful information in the log.

[0026] In some embodiments, the log keyword library 106 can be manually constructed by manually collecting and organizing keywords and phrases according to actual needs and log characteristics. In some embodiments, natural language processing (NLP) or machine learning technology can also be used to automatically extract keywords and phrases from a large amount of log data to construct the log keyword library 106. In addition, the log keyword library 106 can also be established by combining manual construction and automatic extraction methods. One or more embodiments of this specification do not limit the source of the log keyword library 106.

[0027] Normalized coding 108 is an important form to ensure the consistency, readability and maintainability of log data. Normalized coding 108 has the same format, separators, etc. In addition, normalized coding 108 can also have a larger information compression ratio than the log source file while retaining the log valid information as much as possible, thereby saving storage resources. In some embodiments, normalized coding 108 can be used to train a machine learning model to develop a log diagnostic system.

[0028] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present description. The embodiments of the present description may also be applied to other environments with different structures and / or functions.

[0029] The following will combine Figures 2 to 4D The process according to the embodiment of this specification is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It should be understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0030] Figure 2 2 shows a schematic flow chart of a method 200 for log preprocessing according to some embodiments of the present specification. In some embodiments, the method 200 may be performed by, for example Figure 1 The method 200 is implemented by the log aggregation encoder 104 shown. It should be understood that the method 200 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0031] like Figure 2 As shown, at block 210, method 200 may include: determining multiple fields in a log to be processed. Logs are a key component in system management and troubleshooting. A complete log entry usually contains multiple fields to provide enough information to understand and analyze the problem. In various types of logs, different fields are usually separated by delimiters so that tools such as log parsers can accurately identify each field and extract its value.

[0032] In some embodiments, the log aggregation encoder 104 may first determine the delimiter type (e.g., space, colon, comma, etc.) used by the log to be processed. Then, the log aggregation encoder 104 may determine multiple delimiters in the log to be processed based on the delimiter type. Based on these delimiters, the log aggregation encoder 104 may easily determine multiple fields between the delimiters in the log to be processed.

[0033] In some cases, there is no uniform separator in the unordered log, which makes it impossible to smoothly extract the fields in the log for standardized coding. In this case, the log aggregation encoder 104 can rely on a predefined keyword library to complete the division of the fields. The keyword library includes various keywords, phrases or patterns related to the log content, which can match actual valuable words or strings from the log.

[0034] In some embodiments, the log aggregation encoder 104 can first match multiple keywords from the log to be processed based on the keyword library. Then, the log aggregation encoder 104 can determine a field by splicing the current keyword and the string between the current keyword and the next keyword. If the beginning of the log line is a meaningless string, a custom identifier can be used to replace the meaningless string, such as "nonsense". In this way, the log aggregation encoder 104 can separate multiple fields by itself in the absence of a unified separator in the unordered log, so as to perform subsequent standardized encoding operations.

[0035] At block 220, method 200 may include: for each of the plurality of fields, determining a first code associated with the characters of the field and a second code associated with the length of the field. Fuzzy coding is a coding method for reducing the amount of text information while retaining sufficient key features for identification, which may be used to obtain the first code and the second code associated with the log field.

[0036] In some embodiments, the log aggregation encoder 104 can determine one or more keywords from the field, wherein the one or more keywords have actual valuable information. Then, the log aggregation encoder 104 can obtain the first encoding by splicing the one or more keywords. In this way, the first encoding can retain the key character information in the field.

[0037] In some embodiments, the log aggregation encoder 104 can obtain a candidate word set from a field based on a keyword library. Then, the log aggregation encoder 104 can determine the candidate words in the candidate word set whose length is greater than a threshold as keywords. In this way, the amount of text information can be further reduced while retaining key features sufficient for identification.

[0038] In some embodiments, the log aggregation encoder 104 may determine a partitioning granularity associated with the number of digits of the length of the field based on the length of the field. Then, the log aggregation encoder 104 may determine a second encoding based on the partitioning granularity. In this way, the length feature of the field can be retained while reducing the amount of text information for inspection and diagnosis of the log.

[0039] In some embodiments, the log aggregation encoder 104 may determine the divisor of the length of the field based on the number of digits in the length of the field, and then perform an integer division operation on the length of the field based on the divisor. The log aggregation encoder 104 may then determine the result of the integer division operation as a second encoding. For example, for a length of 23, the log aggregation encoder 104 may first determine that the division granularity is a two-digit number. The log aggregation encoder 104 may then determine that the divisor is 10 to obtain an integer division result of "2" as the second encoding. For another example, for a length of 347, the log aggregation encoder 104 may first determine that the division granularity is a three-digit number. The log aggregation encoder 104 may then determine that the divisor is 100 to obtain an integer division result of "3" as the second encoding.

[0040] At block 230, method 200 may include determining a target encoding for the field based on the first encoding and the second encoding. The first encoding includes features associated with the characters of the field, and the second encoding includes features associated with the length of the field. Therefore, the first encoding and the second encoding are combined to include features associated with both the characters and the length of the field.

[0041] In some embodiments, the log aggregation encoder 104 can determine the first code as the prefix of the target code, determine the second code as the suffix of the target code, and then obtain the target code of the field by splicing the prefix and suffix. Finally, the log aggregation encoder 104 can combine the target codes of each field in the log to completely convert the log into a standard format. In this way, the log aggregation encoder 104 can complete the normalization of unordered logs to facilitate the diagnosis and analysis of logs. In some embodiments, the target code can also be used to train a machine learning model to develop a log diagnostic system.

[0042] Figure 3 2 shows a schematic diagram of the structure of a log aggregation encoder 300 according to some embodiments of the present specification. The log aggregation encoder 300 can be regarded as Figure 1 A specific implementation of the log aggregation encoder 104 is shown. Figure 3 As shown, the log aggregation encoder 300 may first obtain an unordered log 302. The unordered log 102 may be a log file that lacks a unified format, has an unclear structure, is difficult to understand and track, and is generated by conflicts generated by multiple people editing at the same time, lack of clear recording specifications, technical limitations, or human errors. Optionally, the unordered log 302 may be read directly from a local device, or from a storage device (such as a database, cloud storage, etc.).

[0043] Then, at step 304, the log aggregation encoder 300 may determine the type of delimiter (e.g., space, colon, comma, etc.) in the unordered log 302. Based on the delimiter type, the log aggregation encoder 300 may determine multiple delimiters in the unordered log 302, thereby determining multiple fields, including field 306-1, field 306-2, ..., field 306-N (collectively referred to as fields 306).

[0044] Then, the log aggregation encoder 300 can obtain a standardized target encoding for each field while retaining the key features of the field. Figure 3 As shown, taking field 306 - 3 as an example, the log aggregation encoder 300 can obtain a first code related to the characters of field 306 - 3 in step 308 , and obtain a second code related to the length of field 306 - 3 in step 310 .

[0045] Specifically, in step 308-1, the log aggregation encoder 300 can match multiple candidate words in field 306-3 based on the log keyword library to obtain a candidate word set. The log keyword library is a predefined vocabulary library containing a large number of meaningful character strings. Then in step 308-2, the log aggregation encoder 300 can further screen the candidate word set to obtain keywords. For example, candidate words with a character length greater than 5 in the candidate word set can be determined as keywords. In step 308-3, the log aggregation encoder 300 can splice multiple keywords together to obtain a first code related to the characters of field 306-3. If there is no qualified keyword as the first code, optionally, a custom keyword can be used to indicate that there is no qualified keyword in the field, such as "nonsense".

[0046] On the other hand, in step 310-1, the log aggregation encoder 300 may first obtain the length of the field 306-3. Then, in step 310-2, the log aggregation encoder 300 may determine the division granularity based on the obtained length. For example, if the length of field 306-3 is 24, then the division granularity may be determined to be 10. For another example, if the length of field 306-3 is 347, then the division granularity may be determined to be 100. In step 310-3, the log aggregation encoder 300 may obtain a second code related to the length of field 306-3 based on the division granularity. For example, if the length of field 306-3 is 24, then the second code may be determined to be the result of 24 divided by 10, that is, "2".

[0047] After obtaining the first code related to the characters of the field and the second code related to the length of the field, in step 312, the log aggregation encoder 300 can use the first code as the prefix of the target code and the second code as the suffix of the target code, so that the first code and the second code can be spliced ​​to obtain the target code for the field. Finally, the log aggregation encoder 300 can combine the target code of each field in the unordered log 302 to complete the normalization of the unordered log 302. In this way, the log format can be normalized while retaining the valid information of the log as much as possible, and the storage occupancy can be reduced.

[0048] Figure 4A-4D A schematic diagram of converting a field into a target code according to some embodiments of the present specification is shown. It is understandable that in one or more embodiments of the specification, "*" is used to represent meaningless characters that do not exist in the log keyword library 106, and the keyword length threshold is preset to 5.

[0049] like Figure 4A As shown, field 401-1 contains the candidate word "inferenceArray" existing in the log keyword library 106 and the meaningless character "*". Since the length of the candidate word "inferenceArray" is 14, which exceeds the preset threshold value 5, the log aggregation encoder 104 can determine the candidate word "inferenceArray" as the keyword of field 401-1. Since the keyword "inferenceArray" is the only keyword in field 401-1, the log aggregation encoder 104 can determine that the first code of field 401-1 is "inferenceArray".

[0050] In addition, since the total length of field 401-1 is 23, log aggregation encoder 104 can determine that the granularity of field 401-1 is divided into two digits. Therefore, log aggregation encoder 104 can divide the total length of field 401-1 by 10, the minimum number of two digits, and use the result "2" as the second code. Then, log aggregation encoder 104 can concatenate the obtained first code and second code to determine that target code 401-2 is "inferenceArray2".

[0051] Therefore, the target code 401 - 2 can normalize the disordered field 401 - 1 in the log while retaining the length characteristics of the disordered field 401 - 1 and the most obvious keyword therein, and has a large information compression ratio, saving storage resources.

[0052] like Figure 4BAs shown, field 402-1 contains candidate words "inferenceArray" and "unit". However, since the length of the candidate word "unit" is 4, which does not meet the preset length threshold, there is still only the unique keyword "inferenceArray" in field 402-1. The log aggregation encoder 104 can determine that the first code of field 402-1 is "inferenceArray". In addition, since the total length of field 402-1 is 33, based on the above-mentioned similar method, the log aggregation encoder 104 can determine that the second code of field 402-1 is "3". Then, the log aggregation encoder 104 can splice the first code and the second code of field 402-1 and determine the target code 402-2 as "inferenceArray3".

[0053] like Figure 4C As shown, field 403-1 contains candidate words "inferenceArray" and "oxford", both of which meet the preset length threshold. Therefore, field 402-1 includes two keywords "inferenceArray" and "oxford". Then, the log aggregation encoder 104 can splice these two keywords together, that is, "inferenceArrayoxford", and determine it as the first code of field 402-1. In addition, since the total length of field 403-1 is 33, based on the above-mentioned similar method, the log aggregation encoder 104 can determine that the second code of field 403-1 is "3". Then, the log aggregation encoder 104 can splice the first code and the second code of field 403-1, and determine the target code 403-2 as "inferenceArrayoxford3".

[0054] like Figure 4D As shown, field 404-1 contains the candidate word "unit", but since the length of the candidate word "unit" is 4, it does not meet the preset length threshold, and therefore field 404-1 does not contain keywords that meet the conditions. In this case, the log aggregation encoder 104 can determine that the first encoding of field 404-1 is "nonsense". In addition, since the total length of field 404-1 is 22, based on the above-mentioned similar method, the log aggregation encoder 104 can determine that the second encoding of field 404-1 is "2". Then, the log aggregation encoder 104 can concatenate the first encoding and the second encoding of field 404-1, and determine that the target encoding 404-2 is "nonsense2".

[0055] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the computing device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0056] Figure 5 FIG. 5 shows a schematic block diagram of a device 500 for log preprocessing according to some embodiments of the present specification. Figure 5 As shown, the apparatus 500 includes: a field determination unit 502 , a fuzzy coding unit 504 and a target coding determination unit 506 .

[0057] In some embodiments, the field determination unit 502 is configured to determine multiple fields in the log to be processed; the fuzzy encoding unit 504 is configured to determine, for each of the multiple fields, a first encoding associated with the characters of the field and a second encoding associated with the length of the field; and the target encoding determination unit 506 is configured to determine the target encoding of the field based on the first encoding and the second encoding.

[0058] In some embodiments, the field determination unit 502 is further configured to determine the delimiter type used by the log to be processed; determine multiple delimiters in the log to be processed based on the delimiter type; and determine multiple fields in the log to be processed based on the multiple delimiters.

[0059] In some embodiments, the fuzzy coding unit 504 is further configured to determine one or more keywords from the field, the one or more keywords having actual valuable information; and obtain the first code by splicing the one or more keywords.

[0060] In some embodiments, the fuzzy coding unit 504 is further configured to obtain a candidate word set from the field based on the keyword library; and determine the candidate words in the candidate word set whose length is greater than a threshold as keywords.

[0061] In some embodiments, the fuzzy coding unit 504 is further configured to determine, based on the length of the field, a partition granularity associated with the number of bits of the length of the field; and determine the second coding based on the partition granularity.

[0062] In some embodiments, the fuzzy coding unit 504 is further configured to determine a divisor of the length of the field based on the number of bits of the length of the field; perform an integer division operation on the length of the field based on the divisor; and determine the result of the integer division operation as a second code.

[0063] In some embodiments, the target encoding determination unit 506 is further configured to determine the first encoding as a prefix of the target encoding; determine the second encoding as a suffix of the target encoding; and obtain the target encoding of the field by concatenating the prefix and the suffix.

[0064] In some embodiments, the field determination unit 502 is further configured to match multiple keywords from the log to be processed based on the keyword library; and determine a field by concatenating the current keyword and the character string between the current keyword and the next keyword.

[0065] In some embodiments, the target encoding is used to train a machine learning model.

[0066] It should be noted that the reference Figures 1 to 4D Further actions or steps shown can be performed by Figure 5 For example, the device 500 may include more modules or units to implement the actions or steps described above, or Figure 5 Some of the units or modules shown may be further configured to implement the actions or steps described above, which will not be repeated here.

[0067] Figure 6 Schematic block diagram of an example device 600 that can be used to implement some embodiments of the present specification is shown. Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or computer program instructions loaded from a storage unit 606 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0068] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0069] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

[0070] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present specification.

[0071] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0073] The computer program instructions for performing the operation of each embodiment of this specification can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, and the programming languages ​​include object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect by the Internet). In certain embodiments, by using the state information of computer-readable program instructions to customize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing the various aspects of this specification.

[0074] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0076] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0078] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for log preprocessing, comprising: Determine multiple fields in the log to be processed; For each field of the plurality of fields, determining a first code associated with characters of the field and a second code associated with a length of the field; as well as Based on the first encoding and the second encoding, a target encoding of the field is determined.

2. The method according to claim 1, wherein determining a plurality of fields in the log to be processed comprises: Determine the delimiter type used by the log to be processed; Based on the delimiter type, determining multiple delimiters in the log to be processed; as well as Based on the multiple delimiters, the multiple fields in the log to be processed are determined.

3. The method of claim 1 , wherein determining a first encoding associated with a character of the field comprises: determining one or more keywords from the field, the one or more keywords having actual valuable information; as well as The first code is obtained by concatenating the one or more keywords.

4. The method of claim 3, wherein determining one or more first character strings from the field comprises: Based on the keyword library, obtaining a candidate word set from the field; as well as The candidate words in the candidate word set whose length is greater than a threshold are determined as the keywords.

5. The method of claim 1 , wherein determining a second encoding associated with the length of the field comprises: Based on the length of the field, determining a partition granularity associated with the number of bits of the length of the field; as well as Based on the division granularity, the second encoding is determined.

6. The method of claim 5, wherein determining the second encoding comprises: Determining a divisor of the length of the field based on the number of bits of the length of the field; Based on the divisor, performing an integer division operation on the length of the field; as well as The result of the integer division operation is determined as the second code.

7. The method according to claim 1, wherein based on the first encoding and the second encoding, obtaining the target encoding of the field comprises: Determine the first code as a prefix of the target code; Determine the second code as a suffix of the target code; as well as The target code of the field is obtained by concatenating the prefix and the suffix.

8. The method according to claim 1, wherein determining a plurality of fields in the log to be processed comprises: Based on the keyword library, matching multiple keywords from the log to be processed; as well as A field is determined by concatenating the current keyword and the character string between the current keyword and the next keyword.

9. The method of claim 1, wherein the target encoding is used to train a machine learning model.

10. A system for log preprocessing, comprising: A field determination unit, configured to determine a plurality of fields in the log to be processed; a fuzzy encoding unit configured to determine, for each of the plurality of fields, a first code associated with a character of the field and a second code associated with a length of the field; as well as The target encoding determination unit is configured to determine a target encoding of the field based on the first encoding and the second encoding.

11. A computing device comprising: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to perform the method as claimed in any one of claims 1 to 9.

12. A computer storage medium comprising machine executable instructions which, when executed by a device, cause the device to perform the method of any one of claims 1 to 9.

13. A computer program product comprising machine executable instructions which, when executed by a device, cause the device to perform the method of any one of claims 1 to 9.

Images