Log processing method, device, equipment and medium

Abstract

The embodiment of the present invention discloses a log processing method, device, equipment and medium. The method includes: applying to a log processing system, obtaining a log to be processed, and performing write verification on the log to be processed to determine the verification result; if the verification result is verification passed, classifying the characters to be processed in the log to be processed according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups; grouping the candidate processing characters according to a preset character grouping symbol to obtain a candidate character group, and iteratively grouping the candidate processing characters in the candidate character group to obtain a reference character group, and determining the character replacement data corresponding to the reference character group, performing character replacement on the reference character group according to the character replacement data, and performing character replacement on the candidate replacement character group according to the character replacement relationship to obtain a processed log corresponding to the log to be processed. The cost of the organization for processing the log to be processed is reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of log processing, and in particular to a log processing method, device, equipment and medium. Background Art

[0002] The current project management software in the construction industry includes various production factors such as man, machine, material, method and environment. Various sensor devices report data very frequently, and the data log size reported by a single type of equipment in a single day ranges from 10M to 5G. Large-capacity disk storage is often required.

[0003] In the existing technology, a single-machine log framework is usually used for storage. A single machine needs a single hard disk with a large capacity to store data. The price of a single large hard disk with the same capacity is often 2-4 times higher than that of a regular hard disk. Alternatively, a distributed log framework is used. The distributed framework supports multiple machines, but there is an additional large memory overhead when querying logs, which often exceeds the memory required by the application system. The above two log processing methods often bring additional costs to the organization. Summary of the invention

[0004] The present invention provides a log processing method, device, equipment and medium, so as to realize the compression processing of logs without changing the original system, thereby reducing the cost of the organization for processing the logs.

[0005] According to one aspect of the present invention, a log processing method is provided, which is applied to a log processing system, comprising:

[0006] Obtaining a log to be processed, and performing a write verification on the log to be processed according to a log writing path and / or a log name format of the log to be processed, and determining a verification result;

[0007] If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups;

[0008] According to a preset character grouping symbol, the candidate processing characters are grouped to obtain a candidate character group, and the candidate processing characters in the candidate character group are iteratively grouped to obtain a reference character group, and character replacement data corresponding to the reference character group is determined;

[0009] According to the character replacement data, character replacement is performed on the reference character group, and according to the character replacement relationship, character replacement is performed on the candidate replacement character group to obtain a processed log corresponding to the log to be processed.

[0010] According to another aspect of the present invention, there is provided a log processing device, which is configured in a log processing system and includes:

[0011] A log verification module is used to obtain a log to be processed, and perform a write verification on the log to be processed according to a log writing path and / or a log name format of the log to be processed, and determine a verification result;

[0012] A character classification module is used for classifying the characters to be processed in the log to be processed according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups if the verification result is that the verification is passed;

[0013] A character grouping module, configured to group the candidate processing characters according to a preset character grouping symbol to obtain a candidate character group, iteratively group the candidate processing characters in the candidate character group to obtain a reference character group, and determine character replacement data corresponding to the reference character group;

[0014] The log processing module is used to perform character replacement on the reference character group according to the character replacement data, and to perform character replacement on the candidate replacement character group according to the character replacement relationship, so as to obtain a processed log corresponding to the log to be processed.

[0015] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When one or more programs are executed by one or more processors, the one or more processors can execute any log processing method provided by the embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any log processing method provided by an embodiment of the present invention when executed.

[0020] An embodiment of the present invention provides a log processing solution, which is applied to a log processing system. The solution obtains a log to be processed, and performs write verification on the log to be processed according to the log writing path and / or log name format of the log to be processed to determine a verification result. If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups. The candidate processing characters are grouped according to a preset character grouping symbol to obtain a candidate character group, and the candidate processing characters in the candidate character group are iteratively grouped to obtain a reference character group, and character replacement data corresponding to the reference character group is determined. According to the character replacement data, the reference character group is replaced with characters, and the candidate replacement character group is replaced with characters according to the character replacement relationship to obtain a processed log corresponding to the log to be processed. The above scheme determines candidate processing characters and candidate replacement character groups from the log to be processed according to the character replacement relationship through the log processing system, and groups the candidate processing characters to obtain a reference character group, performs character replacement on the reference character group according to the character replacement data of the reference character group, and performs character replacement on the candidate replacement character group according to the character replacement relationship. This achieves compression processing of the log to be processed without changing the original system, thereby reducing the cost of the organization for processing the log to be processed. At the same time, by verifying the log to be processed, it is determined whether the log processing system has the authority to process the log to be processed, thereby avoiding invalid processing of the log to be processed by the log processing system and reducing waste of resources.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a log processing method provided by Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a log processing method provided by Embodiment 2 of the present invention;

[0025] Figure 3 It is a structural schematic diagram of a log processing device provided by Embodiment 3 of the present invention;

[0026] Figure 4 It is a structural schematic diagram of an electronic device for implementing a log processing method provided by a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of a log processing method provided in Example 1 of the present invention. This embodiment is applicable to the situation where logs are compressed. The method can be executed by a log processing device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the log processing function.

[0030] See also Figure 1 The log processing method shown is applied to a log processing system, and includes:

[0031] S110: Obtain a log to be processed, and perform a write verification on the log to be processed according to a log writing path and / or a log name format of the log to be processed, and determine a verification result.

[0032] The log processing system refers to a system used to process and store logs. The log processing system is connected to the operating system in communication, and the operating system is connected to the application system in communication. The operating system receives the pending logs sent by the application system. After the log processing system monitors that the operating system has received the pending logs, it obtains the pending logs from the operating system, thereby achieving compression processing of the logs through the introduced log processing system without changing the original system.

[0033] The log to be processed refers to a log that needs to be processed. For example, the log to be processed may be a log that needs to be processed in the construction industry. The log writing path refers to the directory path of the log to be processed. The log name format refers to the format of the log name of the log to be processed.

[0034] In an optional embodiment, a write verification is performed on the log to be processed according to the log write path and / or log name format of the log to be processed, and a verification result is determined, including: matching the log write path of the log to be processed with a preset candidate write path; if the match is successful, using the candidate name format corresponding to the successfully matched candidate write path as the target name format; comparing the log name format of the log to be processed with the target name format, and determining the verification result based on the comparison result.

[0035] The candidate write path refers to the directory path of the log pre-set in the log processing system. Exemplarily, the number of candidate write paths can be multiple. For example, a candidate write path can be D:\system\logs\2024.09.01.log.

[0036] The candidate name format refers to the format of the log name corresponding to the candidate write path. For example, a candidate name format may be *.log wildcard + suffix. The target name format refers to the candidate name format corresponding to the candidate write path that matches the log write path.

[0037] Wherein, a successful match means that there is a candidate write path that is the same as the log write path, and a failed match means that there is no candidate write path that is the same as the log write path.

[0038] The comparison result can be a consistent comparison or an inconsistent comparison. Consistent comparison means that the target name format is the same as the log name format. Inconsistent comparison means that the target name format is different from the log name format.

[0039] The verification result can be used to determine whether the log processing system can process the log to be processed. The verification result can be a pass or a fail. A pass means that the log processing system can process the log to be processed. A fail means that the log processing system cannot process the log to be processed.

[0040] Exemplarily, if the match is successful and the comparison is consistent, the verification result is determined to be a passed verification; if the match is successful and the comparison is inconsistent, the verification result is determined to be a failed verification; if the match fails, the verification result is determined to be a failed verification.

[0041] It should be noted that the log writing path of the log to be processed is matched with the preset candidate writing path. If the match fails, it means that the log processing system cannot process the log to be processed. There is no need to compare the log name format of the log to be processed. At this time, the operating system processes the log to be processed.

[0042] It should be noted that if the verification fails, the operating system processes the log to be processed.

[0043] It can be understood that by verifying the processing authority of the log to be processed according to the candidate write path and the target name format, the log processing system is prevented from processing the logs that cannot be processed, thereby improving the log processing efficiency.

[0044] S120: If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups.

[0045] The character replacement relationship refers to the correspondence between the preset characters and numbers. The characters to be processed refer to the characters in the log to be processed. Exemplarily, the characters to be processed can be Chinese, English or other. The embodiment of the present invention does not specifically limit this.

[0046] The candidate processing characters refer to characters that do not exist in the character replacement relationship, and the candidate replacement character group refers to a character group that exists in the character replacement relationship.

[0047] It should be noted that different character types of characters to be processed may correspond to different character replacement relationships, that is, the character replacement relationship matching the character type may be determined according to the character type of the characters to be processed. For example, if the character type of the characters to be processed is Chinese, the character replacement relationship corresponding to Chinese is used; if the character type of the characters to be processed is English, the character replacement relationship corresponding to English is used.

[0048] Specifically, according to the preset character replacement relationship, the characters to be processed of the log to be processed are divided to obtain candidate processing characters and candidate replacement character groups.

[0049] S130. Grouping candidate processing characters according to a preset character grouping symbol to obtain a candidate character group, iteratively grouping candidate processing characters in the candidate character group to obtain a reference character group, and determining character replacement data corresponding to the reference character group.

[0050] The character grouping symbol refers to a symbol used to initially group candidate processing characters to obtain a candidate character group. The embodiment of the present invention does not impose any limitation on the character grouping symbol, which can be set by a technician based on experience. Exemplarily, the character grouping symbol may include commonly used special symbols such as space, dot, colon, comma, and period.

[0051] It should be noted that in the embodiment of the present invention, when grouping candidate processing characters according to a preset character grouping symbol, it is necessary to add the character grouping symbol to the previous candidate processing character. If the candidate processing characters include ABCD and HF, the aforementioned candidate processing characters are grouped according to "," to obtain two candidate character groups including ABCD and HF. It should be noted that the character grouping symbol may not be considered when determining the character length of the reference character group.

[0052] The candidate character group refers to a character group obtained by grouping candidate processing characters according to character grouping symbols, and the reference character group refers to a character group obtained by iteratively grouping candidate processing characters in the candidate character group.

[0053] It should be noted that if the candidate processing characters are Chinese, when the candidate processing characters in the candidate character group are grouped, they can be split according to the Chinese word segmentation dictionary.

[0054] The character replacement data refers to the data required for character replacement of candidate processing characters in the reference character group. Exemplarily, the character replacement data may include the character index of the reference character group, the candidate processing characters in the reference character group, and the number of occurrences of the reference character group.

[0055] The embodiment of the present invention does not impose any limitation on the presentation form of the character replacement data, which can be set by a technician according to experience or needs. For example, the character replacement data can be presented in a table form.

[0056] The character index of the reference character group refers to the numerical sequence number for sorting the reference character group. The number of occurrences of the reference character refers to the number of occurrences of each candidate processing character in the reference character group as a whole in the log to be processed.

[0057] It should be noted that if the same reference character group exists in the reference character groups corresponding to different candidate character groups, the reference character groups will be merged according to their order of appearance in the log to be processed, and the later reference character group will be merged into the same previous reference character group. It is only necessary to increase the number of occurrences of the reference character group by 1, and there is no need to repeatedly determine the character replacement data of the same reference character group.

[0058] It should be noted that before the candidate processing characters are grouped according to the preset character grouping symbol to obtain the candidate character group, the candidate processing characters can be divided into rows according to the row relationship between the candidate processing characters in the log to be processed; then, the candidate processing characters in each row are grouped according to the preset character grouping symbol.

[0059] S140: Perform character replacement on the reference character group according to the character replacement data, and perform character replacement on the candidate replacement character group according to the character replacement relationship to obtain a processed log corresponding to the log to be processed.

[0060] The processed log refers to the log after all characters to be processed in the log to be processed are replaced with numbers.

[0061] In an optional embodiment, if the character replacement data includes a character index of a reference character group, candidate processing characters in the reference character group, and a number of occurrences of the reference character group, then character replacement is performed on the reference character group according to the character replacement data, including: determining the character index length of the reference character group according to the character index of the reference character group, and determining the character length of the reference character group according to the candidate processing characters in the reference character group; when the sum of the character index length and a preset index value is less than the product of the character length and the number of occurrences of the reference character group, and the number of occurrences of the reference character group is not 0, determining that the reference character group is the character group to be replaced; and replacing the candidate processing characters in the character group to be replaced according to the character index of the character group to be replaced.

[0062] The character index length refers to the number of digits in the character index. The character length refers to the number of candidate processing characters in the reference character group. The character group to be replaced refers to the candidate character group that can be replaced by digits.

[0063] The embodiment of the present invention does not specifically limit the size of the preset index value, which can be set by technicians based on experience or needs. For example, the preset index value can be 1.

[0064] Exemplarily, for any reference character group, if the character index length of the reference character group + 1 < the character length of the reference character group * the number of occurrences of the reference character group, and the number of occurrences of the reference character group is at least once, then the reference character group is determined to be the character group to be replaced.

[0065] Specifically, the candidate processing characters in the to-be-replaced character group are replaced with the character indexes of the to-be-replaced character group.

[0066] It can be understood that by determining whether a reference character group is a character group to be replaced based on the character index length, preset index value, character length and number of occurrences, the reference character group is screened, the number of reference character groups involved in character replacement is reduced, and invalid replacement is avoided.

[0067] In an optional embodiment, after determining that the reference character group is the character group to be replaced, the method further includes: using the difference between the character length of the character group to be replaced and the character index length of the character group to be replaced as the character replacement value of the character group to be replaced; sorting the character group to be replaced according to the character replacement value, and determining the order of character replacement for the character group to be replaced according to the sorting result.

[0068] The character replacement value can be used to determine the order in which the character group to be replaced is replaced. The sorting result refers to the order in which the characters of the character group to be replaced are replaced. Exemplarily, the sorting result can be determined from large to small according to the character replacement value.

[0069] Exemplarily, for any character group to be replaced, the character length of the character group to be replaced minus the character index length of the character group to be replaced = the character replacement value of the character group to be replaced; according to the character replacement value, the character groups to be replaced are sorted from large to small to obtain a sorting result.

[0070] It can be understood that by introducing character replacement values ​​and determining the order of character replacement for the character group to be replaced according to the character replacement values, it is possible to use one character index to replace as many candidate processing characters as possible, thereby achieving greater compression of candidate processing characters.

[0071] In an optional embodiment, performing character replacement on the candidate replacement character group according to the character replacement relationship includes: determining the character replacement number corresponding to the candidate replacement character group according to the character replacement relationship, and performing character replacement on the candidate replacement character group according to the character replacement number.

[0072] The character replacement number refers to the number corresponding to the candidate replacement character group recorded in the character replacement relationship.

[0073] It can be understood that by performing character replacement on the candidate replacement character group according to the character replacement numbers corresponding to the candidate replacement character group in the character replacement relationship, the accuracy of the character replacement is ensured.

[0074] In the embodiment of the present invention, the character group to be replaced can be screened according to the candidate replacement character group to obtain the target replacement character group, and the character replacement relationship can be updated according to the target replacement character group and the corresponding character index. The target replacement character group refers to the character group to be replaced that is complementary to the candidate replacement character group.

[0075] It can be understood that by introducing the target replacement character group, at least partial repeated replacement is avoided, resource waste is reduced, and character replacement efficiency is improved; at the same time, the character replacement relationship is updated, improving the real-time and comprehensiveness of the character replacement relationship.

[0076] In the embodiment of the present invention, a digital separator symbol may be introduced to divide the numbers representing different character groups in the processed log. The digital separator symbol may be used to divide the numbers of the replacement character group. The embodiment of the present invention does not impose any restrictions on the digital separator symbol, which may be set by a technician based on experience. Exemplarily, the digital separator symbol may be / .

[0077] For example, after symbol replacement for any part of the characters to be processed in the log to be processed, 1\29\31\33\35\36\12 is obtained.

[0078] An embodiment of the present invention provides a log processing solution, which is applied to a log processing system. The solution obtains a log to be processed, and performs write verification on the log to be processed according to the log writing path and / or log name format of the log to be processed to determine a verification result. If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups. The candidate processing characters are grouped according to a preset character grouping symbol to obtain a candidate character group, and the candidate processing characters in the candidate character group are iteratively grouped to obtain a reference character group, and character replacement data corresponding to the reference character group is determined. According to the character replacement data, the reference character group is replaced with characters, and the candidate replacement character group is replaced with characters according to the character replacement relationship to obtain a processed log corresponding to the log to be processed. The above scheme determines candidate processing characters and candidate replacement character groups from the log to be processed according to the character replacement relationship through the log processing system, and groups the candidate processing characters to obtain a reference character group, performs character replacement on the reference character group according to the character replacement data of the reference character group, and performs character replacement on the candidate replacement character group according to the character replacement relationship. This achieves compression processing of the log to be processed without changing the original system, thereby reducing the cost of the organization for processing the log to be processed. At the same time, by verifying the log to be processed, it is determined whether the log processing system has the authority to process the log to be processed, thereby avoiding invalid processing of the log to be processed by the log processing system and reducing waste of resources.

[0079] Based on the above technical solutions, the log processing system provides an algorithm interface, which can determine whether it is necessary to load some implemented string compression algorithms before using the algorithm. If it is necessary to load a third-party algorithm, such as Huffman Coding, the log processing system will call the third-party algorithm before compression and will also use the third-party algorithm when decompressing.

[0080] The log processing system of the embodiment of the present invention can use the system monitoring event API (Application Programming Interface) to judge, and can use methods including but not limited to file filter driver and kernel module programming to realize the classification of characters to be processed in the log to be processed. The specific implementation needs to be determined according to the API, driver and security module provided by the specific operating system.

[0081] Embodiment 2

[0082] Figure 2It is a flow chart of a log processing method provided by the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the operation of "iteratively grouping the candidate processing characters in the candidate character group to obtain a reference character group" into "determining the current number of groups for the current iterative grouping of the candidate processing characters in the candidate character group; according to the current number of groups, performing the current iterative grouping of the candidate processing characters in the candidate character group to obtain a reference character group including the current number of candidate processing characters", so as to improve the determination mechanism of the reference character group. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0083] See also Figure 2 The log processing method shown includes:

[0084] S210: Obtain a log to be processed, and perform write verification on the log to be processed according to the log writing path and / or log name format of the log to be processed, and determine a verification result.

[0085] S220: If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups.

[0086] S230: Group candidate processing characters according to a preset character grouping symbol to obtain a candidate character group.

[0087] S240: Determine the current number of groups for the current iteration of grouping the candidate processing characters in the candidate character group.

[0088] The current iterative grouping refers to the grouping performed this time. The current grouping quantity refers to the quantity of candidate processing characters in the reference character group. Exemplarily, the current grouping quantity may be increased one by one.

[0089] S250, performing current iteration grouping on the candidate processing characters in the candidate character group according to the current grouping number, obtaining a reference character group including the candidate processing characters of the current grouping number, and determining character replacement data corresponding to the reference character group.

[0090] Specifically, according to the current grouping number, the candidate processing characters in the candidate character group are grouped in the current iteration to obtain a reference character group corresponding to the candidate character group and including the candidate processing characters of the current grouping number.

[0091] It should be noted that, if the number of candidate processing characters in the reference character group is at least two, it is necessary to ensure that the at least two candidate processing characters are adjacent characters.

[0092] In an optional embodiment, when the character length of the reference character group reaches a preset group length threshold, or the reference character group is identical to the corresponding candidate character group, the iterative grouping is stopped.

[0093] The embodiment of the present invention does not impose any limitation on the size of the preset group length threshold, which can be set by technicians based on experience or determined repeatedly through a large number of experiments. It should be noted that one candidate character group can correspond to multiple reference character groups.

[0094] Exemplarily, for any candidate character group, if the character length of the candidate character group is greater than or equal to a preset grouping length threshold, then when the character length of the candidate processing characters in any reference character group corresponding to the candidate character group reaches the preset grouping length threshold, the iterative grouping of the candidate character group is stopped.

[0095] Exemplarily, for any candidate character group, if the character length of the candidate processing character in the candidate character group is less than a preset grouping length threshold, then when any reference character group corresponding to the candidate character group is the same as the candidate character group, stop iterating the grouping of the candidate characters.

[0096] Specifically, for any candidate character group, when the character length of the reference character group corresponding to the candidate character group reaches a preset grouping length threshold, stop performing the next iterative grouping of the candidate character group; or, when any reference character group corresponding to the candidate character group is the same as the candidate character group, stop performing the next iterative grouping of the candidate character group.

[0097] Optionally, determine the character length of the candidate processing characters in the candidate character group; select a target iteration stop condition from the candidate iteration stop conditions based on the character length and a preset grouping length threshold; when the reference character group corresponding to the candidate sub-character group meets the target iteration stop condition, stop the next iteration grouping of the candidate character group.

[0098] The candidate iteration stop condition refers to the condition for stopping iterative grouping of candidate character groups. Exemplarily, the candidate iteration stop condition may include that the character length of the reference character group reaches a preset grouping length threshold, and the reference character group is the same as the corresponding candidate character group. The target iteration stop condition refers to the condition for stopping iterative grouping of a certain candidate character group.

[0099] Exemplarily, if the character length of the candidate character group is greater than or equal to a preset group length threshold, then the target iteration stopping condition for the candidate character group is that the character length of the reference character group reaches the preset group length threshold; if the character length of the candidate character group is less than the preset group length threshold, then the target iteration stopping condition for the candidate character group is that the reference character group is the same as the corresponding candidate character group.

[0100] It should be noted that when the character length of the reference character group reaches the preset grouping length threshold, all candidate character groups must stop the next iterative grouping, which means that the iterative grouping can be stopped globally; when the reference character group of any candidate character group is the same as the candidate character group, the candidate character group stops the next iterative grouping, which means that the iterative grouping can be stopped locally.

[0101] It can be understood that by stopping iterative grouping when the character length of the reference character group reaches a preset grouping length threshold, or the reference character group is identical to the corresponding candidate character group, the accuracy and timeliness of stopping iterative grouping are improved.

[0102] S260: Perform character replacement on the reference character group according to the character replacement data, and perform character replacement on the candidate replacement character group according to the character replacement relationship to obtain a processed log corresponding to the log to be processed.

[0103] The embodiment of the present invention can also process the existing and uncompressed logs in the log processing system. For example, according to the monitoring directory in the log processing system, the logs corresponding to the monitoring directory are traversed to determine the uncompressed logs, and the uncompressed logs are used as the logs to be processed. Subsequently, the compression of the uncompressed logs is realized through the method of this solution. The monitoring directory can be used to record the log identifiers in the log processing system to facilitate the search of logs.

[0104] The embodiment of the present invention provides a log processing scheme, which refines the operation of iteratively grouping candidate processing characters in a candidate character group to obtain a reference character group, and then determines the current number of groups for the current iterative grouping of candidate processing characters in the candidate character group; and performs the current iterative grouping of candidate processing characters in the candidate character group according to the current number of groups to obtain a reference character group including the current number of candidate processing characters, thereby improving the determination mechanism of the reference character group. The above scheme improves the comprehensiveness of the reference character group and avoids the loss of the reference character group by introducing the current number of groups and performing the current iterative grouping of candidate processing characters in the candidate character group according to the current number of groups to obtain a reference character group.

[0105] Based on the above technical solution, the embodiment of the present invention also provides a method for log restoration. Log restoration is essentially the reverse processing of log compression, that is, after determining the target processed log, the compression algorithm, indexing, decompression, and replacement of compressed content are used to output the original log content to the application system.

[0106] Exemplarily, the application system sends a log reading instruction to the operating system, the log processing system listens to the log reading instruction, and in response to the log reading instruction, determines whether there is a target processed log among the candidate processed logs stored locally; if so, the target processed log is restored according to the character replacement relationship to obtain a target log that matches the log reading instruction, and the target log is fed back to the application system through the operating system.

[0107] The log reading instruction refers to a command for obtaining a target log. Exemplarily, the log reading instruction may include a log identifier of the target log. The log identifier refers to an identity used to characterize the target log. Exemplarily, the log identifier may be a log name.

[0108] The target log refers to the log required by the application system. The candidate processed log refers to the compressed log stored in the log processing system. The target processed log refers to the candidate processed log that matches the log reading instruction.

[0109] It should be noted that the number separators need to be removed when restoring characters.

[0110] For example, if there is no processed log matching the log reading instruction in the log processing system, it indicates that the log required by the application system is stored in the operating system. The operating system searches for the target log matching the log reading instruction locally in response to the log reading instruction, and feeds the target log back to the application system.

[0111] The technical solution provided by the embodiment of the present invention utilizes the bottom layer of the operating system to construct a log processing system for file writing, reading and monitoring, and reconstructs, converts and compresses the log by monitoring the writing. For log files with a lot of repeated content, the capacity can be greatly compressed. Without modifying the application, a large amount of cost can be saved for the organization. At the same time, the default compression algorithm can be replaced, without limiting any one, and there are unlimited possibilities.

[0112] Embodiment 3

[0113] Figure 3 This is a structural diagram of a log processing device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of compressing logs, and the method can be executed by a log processing device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the log processing function.

[0114] like Figure 3 As shown, the device includes: a log verification module 310, a character classification module 320, a character grouping module 330 and a log processing module 340. Among them,

[0115] The log verification module 310 is used to obtain the log to be processed, and perform write verification on the log to be processed according to the log writing path and / or log name format of the log to be processed, and determine the verification result;

[0116] The character classification module 320 is used to classify the characters to be processed in the log to be processed according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups if the verification result is that the verification is passed;

[0117] A character grouping module 330, configured to group the candidate processing characters according to a preset character grouping symbol to obtain a candidate character group, iteratively group the candidate processing characters in the candidate character group to obtain a reference character group, and determine character replacement data corresponding to the reference character group;

[0118] The log processing module 340 is used to perform character replacement on the reference character group according to the character replacement data, and to perform character replacement on the candidate replacement character group according to the character replacement relationship, so as to obtain a processed log corresponding to the log to be processed.

[0119] An embodiment of the present invention provides a log processing solution, which is applied to a log processing system. The solution obtains a log to be processed, and performs write verification on the log to be processed according to the log writing path and / or log name format of the log to be processed to determine a verification result. If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to a preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups. The candidate processing characters are grouped according to a preset character grouping symbol to obtain a candidate character group, and the candidate processing characters in the candidate character group are iteratively grouped to obtain a reference character group, and character replacement data corresponding to the reference character group is determined. According to the character replacement data, the reference character group is replaced with characters, and the candidate replacement character group is replaced with characters according to the character replacement relationship to obtain a processed log corresponding to the log to be processed. The above scheme determines candidate processing characters and candidate replacement character groups from the log to be processed according to the character replacement relationship through the log processing system, and groups the candidate processing characters to obtain a reference character group, performs character replacement on the reference character group according to the character replacement data of the reference character group, and performs character replacement on the candidate replacement character group according to the character replacement relationship. This achieves compression processing of the log to be processed without changing the original system, thereby reducing the cost of the organization for processing the log to be processed. At the same time, by verifying the log to be processed, it is determined whether the log processing system has the authority to process the log to be processed, thereby avoiding invalid processing of the log to be processed by the log processing system and reducing waste of resources.

[0120] Optionally, the character grouping module 330 includes:

[0121] A grouping number determining unit, used to determine the current number of groups for performing current iteration grouping on the candidate processing characters in the candidate character group;

[0122] The reference character group determination unit is used to perform current iterative grouping on the candidate processing characters in the candidate character group according to the current grouping quantity, so as to obtain a reference character group including the candidate processing characters of the current grouping quantity.

[0123] Optionally, when the character length of the reference character group reaches a preset group length threshold, or the reference character group is identical to the corresponding candidate character group, the iterative grouping is stopped.

[0124] Optionally, if the character replacement data includes a character index of a reference character group, a candidate processing character in the reference character group, and the number of occurrences of the reference character group, the log processing module 340 includes:

[0125] a length determining unit, configured to determine a character index length of the reference character group according to a character index of the reference character group, and to determine a character length of the reference character group according to a candidate processing character in the reference character group;

[0126] a to-be-replaced character group determination unit, configured to determine that the reference character group is a to-be-replaced character group when the sum of the character index length and the preset index value is less than the product of the character length and the number of occurrences of the reference character group, and the number of occurrences of the reference character group is not 0;

[0127] The character replacement unit is used to replace the candidate processing characters in the character group to be replaced according to the character index of the character group to be replaced.

[0128] Optionally, after determining that the reference character group is a character group to be replaced, the device further includes:

[0129] a character replacement value determining unit, configured to use a difference between a character length of the to-be-replaced character group and a character index length of the to-be-replaced character group as a character replacement value of the to-be-replaced character group;

[0130] The character replacement order determining unit is used to sort the to-be-replaced character group according to the character replacement value, and determine the order of character replacement for the to-be-replaced character group according to the sorting result.

[0131] Optionally, the log verification module 310 includes:

[0132] A path matching unit, used for matching the log writing path of the log to be processed with a preset candidate writing path;

[0133] a target name format determination unit, configured to, if the match is successful, use the candidate name format corresponding to the successfully matched candidate write path as the target name format;

[0134] The verification result determination unit is used to compare the log name format of the log to be processed with the target name format, and determine the verification result according to the comparison result.

[0135] Optionally, the log processing module 340 includes:

[0136] The candidate character replacement unit is used to determine the character replacement number corresponding to the candidate replacement character group according to the character replacement relationship, and perform character replacement on the candidate replacement character group according to the character replacement number.

[0137] The log processing device provided in the embodiment of the present invention can execute the log processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each log processing method.

[0138] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of the pending logs and candidate write paths involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0139] Embodiment 4

[0140] Figure 4 4 is a schematic diagram of the structure of an electronic device for implementing a log processing method provided by Embodiment 4 of the present invention. The electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0141] like Figure 4As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0142] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 executes the various methods and processes described above, such as a log processing method.

[0144] In some embodiments, the log processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the log processing method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to execute the log processing method in any other appropriate manner (e.g., by means of firmware).

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log processing method, applied to a log processing system, characterized in that: include: Obtaining a log to be processed, and performing a write verification on the log to be processed according to a log writing path and / or a log name format of the log to be processed, and determining a verification result; If the verification result is that the verification is passed, the characters to be processed in the log to be processed are classified according to the preset character replacement relationship to obtain candidate processing characters and candidate replacement character groups; wherein the candidate processing characters refer to characters that do not exist in the character replacement relationship; and the candidate replacement character group refers to a character group that exists in the character replacement relationship; According to a preset character grouping symbol, the candidate processing characters are grouped to obtain a candidate character group, and the candidate processing characters in the candidate character group are iteratively grouped to obtain a reference character group, and character replacement data corresponding to the reference character group is determined; According to the character replacement data, character replacement is performed on the reference character group, and according to the character replacement relationship, character replacement is performed on the candidate replacement character group to obtain a processed log corresponding to the log to be processed.

2. The method according to claim 1, characterized in that: The iterative grouping of candidate processing characters in the candidate character group to obtain a reference character group includes: Determining the current number of groups for performing current iteration grouping of candidate processing characters in the candidate character group; According to the current grouping number, the candidate processing characters in the candidate character group are grouped in current iteration to obtain a reference character group including the candidate processing characters of the current grouping number.

3. The method according to claim 2, characterized in that When the character length of the reference character group reaches a preset group length threshold, or the reference character group is identical to the corresponding candidate character group, the iterative grouping is stopped.

4. The method according to claim 1, characterized in that: If the character replacement data includes a character index of a reference character group, a candidate processing character in the reference character group, and an occurrence count of the reference character group, then performing character replacement on the reference character group according to the character replacement data includes: Determining the character index length of the reference character group according to the character index of the reference character group, and determining the character length of the reference character group according to the candidate processing characters in the reference character group; When the sum of the character index length and the preset index value is less than the product of the character length and the number of occurrences of the reference character group, and the number of occurrences of the reference character group is not 0, determining that the reference character group is the character group to be replaced; The candidate processing characters in the to-be-replaced character group are replaced according to the character index of the to-be-replaced character group.

5. The method according to claim 4, characterized in that After determining that the reference character group is the character group to be replaced, the method further includes: Using the difference between the character length of the to-be-replaced character group and the character index length of the to-be-replaced character group as the character replacement value of the to-be-replaced character group; The to-be-replaced character groups are sorted according to the character replacement values, and the order of character replacement for the to-be-replaced character groups is determined according to the sorting result.

6. The method according to claim 1, characterized in that The step of performing write verification on the log to be processed according to the log write path and / or log name format of the log to be processed and determining the verification result includes: Matching the log writing path of the log to be processed with a preset candidate writing path; If the match is successful, the candidate name format corresponding to the candidate write path that is successfully matched is used as the target name format; The log name format of the log to be processed is compared with the target name format, and a verification result is determined according to the comparison result.

7. The method according to claim 1, characterized in that The step of performing character replacement on the candidate replacement character group according to the character replacement relationship includes: According to the character replacement relationship, the character replacement number corresponding to the candidate replacement character group is determined, and the character replacement of the candidate replacement character group is performed according to the character replacement number.

8. A log processing device, configured in a log processing system, characterized in that: include: A log verification module is used to obtain a log to be processed, and perform a write verification on the log to be processed according to a log writing path and / or a log name format of the log to be processed, and determine a verification result; A character classification module is used for classifying the characters to be processed in the log to be processed according to a preset character replacement relationship if the verification result is passed, and obtaining candidate processing characters and candidate replacement character groups; wherein the candidate processing characters refer to characters that do not exist in the character replacement relationship; and the candidate replacement character group refers to a character group that exists in the character replacement relationship; A character grouping module, configured to group the candidate processing characters according to a preset character grouping symbol to obtain a candidate character group, iteratively group the candidate processing characters in the candidate character group to obtain a reference character group, and determine character replacement data corresponding to the reference character group; The log processing module is used to perform character replacement on the reference character group according to the character replacement data, and to perform character replacement on the candidate replacement character group according to the character replacement relationship, so as to obtain a processed log corresponding to the log to be processed.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a log processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a log processing method as described in any one of claims 1 to 7 is implemented.

Images