Log information compression method and device, equipment, storage medium and program product

Abstract

The invention provides a log information compression method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring to-be-compressed log information; performing grouping processing on the to-be-compressed log information to obtain at least one log information group; each log information group comprises a plurality of pieces of to-be-compressed log information; obtaining a pre-stored log template; performing matching processing on the log information group according to a pre-stored log template; if the matching is successful, determining a target log template according to a matching result; if the matching is not successful, performing analysis processing on the log information group based on a large language model to obtain a target log template; and according to the target log template, performing compression processing on the log information group to obtain corresponding compressed log information. Efficient and accurate analysis of log data can be realized, so that the error rate and efficiency of log information compression are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for compressing log information. Background Technology

[0002] As the system grows exponentially, the amount of log data generated by its components also explodes. The storage cost of these logs rises sharply due to high redundancy, necessitating the operations team to reduce storage pressure and improve log analysis efficiency through log compression technology.

[0003] Current log compression technology mainly extracts key log information through information matching methods, which primarily involves manually creating templates to extract key log information.

[0004] However, template writing relies on human experience and is difficult to adapt to the diversity of log formats, resulting in a large error rate and low efficiency in log information compression. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for compressing log information, in order to solve the technical problems of large error rate and low efficiency in the compression of log information in the prior art.

[0006] Firstly, this application provides a method for compressing log information, including:

[0007] Obtain the log information to be compressed;

[0008] The log information to be compressed is grouped to obtain at least one log information group; each log information group includes multiple log information to be compressed.

[0009] Obtain a pre-stored log template; wherein the pre-stored log template is generated by the large language model parsing multiple log information sample groups;

[0010] The log information group is matched according to the pre-stored log template;

[0011] If a match is found, the target log template is determined based on the matching result;

[0012] If the match fails, the log information group is parsed based on the large language model to obtain the target log template;

[0013] Based on the target log template, the log information group is compressed to obtain the corresponding compressed log information.

[0014] In one possible implementation, the step of grouping the log information to be compressed to obtain at least one log information group includes: determining corresponding grouping information based on the log information to be compressed; wherein the grouping information includes at least one of log length, K prefix tag, and tag string; and grouping the log information to be compressed based on the grouping information to obtain at least one log information group.

[0015] In one possible implementation, the step of grouping the log information to be compressed according to the grouping information to obtain at least one log information group includes: performing a first-level grouping process on the log information to be compressed according to the log length to obtain a first grouping result; performing a second-level grouping process on the first grouping result according to the K-prefix tag to obtain a second grouping result; calculating the tag string similarity of the second grouping result according to the tag string; and performing a third-level grouping process on the second grouping result according to the tag string similarity to obtain at least one log information group.

[0016] In one possible implementation, before obtaining the pre-stored log template, the method further includes: obtaining multiple log information sample groups; wherein the log information sample groups include multiple log information samples; filtering the log information sample groups to obtain multiple key log information samples; inputting the multiple key log information samples into a large language model, so that the large language model outputs an initial log template based on preset prompt word examples; optimizing the initial log template through a self-reflection mechanism to obtain a log template; and storing the log template.

[0017] In one possible implementation, the step of filtering the log information sample group to obtain multiple key log information samples includes: using a retrieval enhancement generation algorithm based on Tanimoto similarity to iteratively filter the log information samples in the log information sample group to obtain multiple key log information samples.

[0018] In one possible implementation, the step of optimizing the initial log template through a self-reflection mechanism to obtain a log template includes: matching all log information samples in the log information sample group according to the initial log template; if the initial log template can match all log information samples, then the initial log template is determined as the log template; if there are any log information samples that do not match the initial log template, then a new log template is generated again using a large language model and iteratively optimized until a log template that can match all log information samples is obtained.

[0019] In one possible implementation, after parsing the log information group based on a large language model to obtain the target log template if the matching fails, the method further includes: optimizing the target log template through a self-reflection mechanism; and storing the optimized target log template.

[0020] In one possible implementation, the method further includes updating the pre-stored log template in response to a user's log template update operation.

[0021] Secondly, this application provides a log information compression device, comprising:

[0022] The first acquisition module is used to acquire the log information to be compressed;

[0023] The first processing module is used to group the log information to be compressed to obtain at least one log information group; each log information group includes multiple log information to be compressed.

[0024] The second acquisition module is used to acquire a pre-stored log template; wherein the pre-stored log template is generated by the large language model parsing multiple log information sample groups;

[0025] The second processing module is used to perform matching processing on the log information group according to the pre-stored log template;

[0026] The determination module is used to determine the target log template based on the matching result if a match is successful.

[0027] The third processing module is used to parse the log information group based on the large language model to obtain the target log template if the matching fails.

[0028] The fourth processing module is used to compress the log information group according to the target log template to obtain the corresponding compressed log information.

[0029] Thirdly, this application provides a log information compression device, including: a memory and a processor;

[0030] The memory stores computer-executed instructions;

[0031] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0032] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0033] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0034] The log information compression method, apparatus, device, storage medium, and program product provided in this application group log information, then use a large language model to parse the log information groups to obtain log templates, and improve the efficiency of log template extraction by pre-storing templates. This enables efficient and accurate parsing of log data, thereby improving the error rate and efficiency of log information compression. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] Figure 1 A schematic diagram illustrating a scenario for the log information compression method provided in this application embodiment;

[0037] Figure 2 A flowchart illustrating a method for compressing log information according to an embodiment of this application;

[0038] Figure 3 A flowchart illustrating another log information compression method provided in this application embodiment;

[0039] Figure 4 A schematic diagram of the structure of the log information compression device provided in the embodiments of this application;

[0040] Figure 5 This is a schematic diagram of the structure of the log information compression device provided in the embodiments of this application.

[0041] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0044] Currently, in log information compression technology, the method of manually creating log templates to extract key log information is widely used. Logs are unstructured data, and log information is generally lengthy and has a certain format. For unstructured text data with a certain format, log templates are usually set up based on empirical rules, and then the templates are used to extract static text with a fixed format and dynamic variables as key log information. However, template creation relies on human experience and is difficult to adapt to the diversity of log formats, resulting in a large error rate and low efficiency in log information compression.

[0045] The log information compression method provided in this application groups the log information, then uses a large language model to parse the log information groups to obtain log templates, and improves the efficiency of log template extraction by pre-storing templates. This enables efficient and accurate parsing of log data, thereby improving the error rate and efficiency of log information compression.

[0046] Figure 1 This is a schematic diagram illustrating a scenario for the log information compression method provided in an embodiment of this application, such as... Figure 1 As shown, the scene is a computer device, including: a receiving device 101, a processor 102, and a display device 103.

[0047] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the method for compressing log information. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0048] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, which can obtain the log information to be compressed.

[0049] The processor 102 can process the log information to be compressed in order to determine the corresponding compressed log information.

[0050] The display device 103 can be used to display the aforementioned compressed log information, etc.

[0051] The display device can also be a touch screen, used to receive user commands while displaying the above content, so as to achieve interaction with the user.

[0052] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0053] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0055] Figure 2 This is a flowchart illustrating a log information compression method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 The computer equipment shown is not specifically limited in this embodiment. Figure 2 As shown, the method includes:

[0056] S201: Obtain the log information to be compressed.

[0057] S202: The log information to be compressed is grouped to obtain at least one log information group; each log information group includes multiple log information to be compressed.

[0058] Optionally, based on the log information to be compressed, corresponding grouping information is determined; wherein, the grouping information includes at least one of log length, K prefix tag and tag string; based on the grouping information, the log information to be compressed is grouped to obtain at least one log information group.

[0059] Grouping information allows for precise grouping, improves template matching accuracy, and ultimately enhances log compression accuracy.

[0060] Optionally, the log information to be compressed is grouped according to the grouping information to obtain at least one log information group, including: performing a first-level grouping process on the log information to be compressed according to the log length to obtain a first grouping result; performing a second-level grouping process on the first grouping result according to the K prefix tag to obtain a second grouping result; calculating the tag string similarity of the second grouping result according to the tag string; and performing a third-level grouping process on the second grouping result according to the tag string similarity to obtain at least one log information group.

[0061] Optionally, the log length can be the number of strings in the log after being split by spaces.

[0062] Optionally, before performing second-level grouping processing on the first grouping result based on the K prefix mark to obtain the second grouping result, the method further includes: obtaining the entropy value of the string of log information to be compressed; and determining the value of K based on the entropy value of the string using a preset dynamic adjustment rule.

[0063] Optionally, the K in the K prefix can take the value 2 or 3, etc.

[0064] Optionally, the first grouping result is subjected to a second-level grouping process based on the K-prefix marker to obtain the second grouping result, including: taking the first K strings of the log information as the K-prefix marker; and performing a second-level grouping process on the first grouping result based on the K-prefix marker to obtain the second grouping result.

[0065] Optionally, the second grouping results can be stored in the form of a fixed-depth tree, where if K is 3, the root node of the fixed-depth tree represents a length group in a specific first grouping result. The first-level child nodes represent the value of the first tag of the K-prefix. The second-level child nodes, under the first-level child nodes, represent the value of the second tag of the K-prefix. The third-level child nodes, under the second-level child nodes, represent the value of the third tag of the K-prefix.

[0066] Optionally, based on the similarity of the tag strings, the second grouping results are subjected to a third-level grouping process to obtain at least one log information group, including: determining a log template for each log information group in the second grouping results; sequentially calculating the similarity between the log information and the log template of each log information group; and performing a third-level grouping process on the log information based on a preset similarity threshold and the similarity corresponding to the log information to obtain at least one log information group.

[0067] Optionally, the formula for calculating the similarity between log information and the log template of each log information group is: Similarity = Number of common strings / Total number of log strings. Wherein, the number of common strings is the number of strings in the log information and the log template that are identical in content at the same position; the total number of log strings is the total number of strings in the log information.

[0068] It should be noted that in the second grouping results, the total number of strings in the log information is the same as the total number of strings in the log template.

[0069] Optionally, the log information is grouped at the third level based on a preset similarity threshold and the similarity of the log information, including: if the similarity of the log information is not less than the preset similarity threshold, the log is grouped into the log information group corresponding to the log template; if the similarity of the log information is less than the preset similarity threshold, a new log information group is created and the fixed depth tree is dynamically updated.

[0070] By grouping log information based on similarity and dynamically updating the fixed-depth tree, it is possible to ensure that log information groups evolve as incoming log information develops, thereby continuously optimizing the accuracy and efficiency of the log grouping process.

[0071] Optionally, the preset similarity threshold can be 0.6.

[0072] By employing a grouping strategy based on three dimensions—log length, K-prefix tagging, and tag string similarity—the computational complexity of the grouping stage can be significantly reduced while ensuring grouping accuracy. Length-based grouping reduces computational complexity, K-prefix tagging improves grouping efficiency, and string similarity grouping ensures semantic consistency within groups, ultimately achieving efficient and accurate log grouping.

[0073] S203: Obtain the pre-stored log template; wherein, the pre-stored log template is generated by the large language model from parsing multiple log information sample groups.

[0074] Optionally, before obtaining the pre-stored log template, the method further includes: obtaining multiple log information sample groups; wherein each log information sample group includes multiple log information samples; filtering the log information sample groups to obtain multiple key log information samples; inputting the multiple key log information samples into a large language model so that the large language model outputs an initial log template based on preset prompt word examples; optimizing the initial log template through a self-reflection mechanism to obtain a log template; and storing the log template.

[0075] It should be noted that at least a preset number of log information samples should be selected for filtering in each log information sample group. If the number of log information samples in a log information sample group is less than the preset number, all log information samples should be selected for filtering.

[0076] The preset quantity can be 300.

[0077] It should be noted that the preset prompt examples are used to standardize the input / output format, and these prompt examples remain unchanged across all systems.

[0078] By obtaining log templates through key log information samples and a self-reflection mechanism, the accuracy of log templates can be improved, thereby increasing the accuracy and efficiency of log parsing.

[0079] Optionally, the log information sample group is filtered to obtain multiple key log information samples, including: using a retrieval enhancement generation algorithm based on Tanimoto similarity to iteratively filter the log information samples in the log information sample group to obtain multiple key log information samples.

[0080] By using a retrieval enhancement generation algorithm based on Tanimoto similarity, multiple key log information samples are obtained, which can improve the accuracy of log templates.

[0081] Optionally, Tanimoto similarity is defined as the ratio of the size of the intersection to the size of the union of two sets.

[0082] Optionally, a retrieval enhancement generation algorithm based on Tanimoto similarity is used to iteratively filter the log information samples in the log information sample group to obtain multiple key log information samples, including Sa~Sf:

[0083] Sa: Select the log information sample with the longest character count from the log information sample group as the initial reference sample.

[0084] Sb: Calculate the first Tanimoto similarity between the initial reference sample and other log information samples in the log information sample group in turn.

[0085] Sc: The log information sample with the lowest Tanimoto similarity is identified as the key log information sample and added to the key log information sample set.

[0086] Sd: Calculate the second Tanimoto similarity between the key log information sample set and other log information samples in the log information sample group that were not added to the key log information sample set.

[0087] Optionally, for each log information sample in the other log information samples, calculate one or more Tanimoto similarities between the log information sample and the key log information samples in the key log information sample set; determine the lowest Tanimoto similarity among the one or more Tanimoto similarities as the second Tanimoto similarity.

[0088] Se: The log information sample with the lowest similarity to the second Tanimoto is identified as the key log information sample and added to the key log information sample set.

[0089] Sf: Repeat steps Sd and Se until the number of key log information samples in the key log information sample set reaches the preset number threshold to obtain multiple key log information samples.

[0090] Based on Tanimoto similarity, log information samples in the log information sample group are iteratively filtered to obtain key log information samples, ensuring that the key log information samples effectively represent the commonalities and diversity in the log information sample group.

[0091] Optionally, the preset quantity threshold can be 3.

[0092] Optionally, step Sb includes: splitting the log information samples to obtain a set of phrases corresponding to each log information sample; and sequentially calculating the first Tanimoto similarity between the phrase set of the initial reference sample and the phrase sets of other log information samples in the log information sample group.

[0093] Optionally, the initial log template is optimized through a self-reflection mechanism to obtain a log template, including: matching all log information samples in the log information sample group according to the initial log template; if the initial log template can match all log information samples, then the initial log template is determined as the log template; if there is any log information sample that does not match the initial log template, then a new log template is generated again using the large language model and iteratively optimized until a log template that can match all log information samples is obtained.

[0094] By performing sample matching on the initial log template and iteratively optimizing it through a large language model, the automatic optimization of the log template can be achieved, which can improve the adaptability of the log template while reducing manual maintenance costs.

[0095] S204: Match the log information groups according to the pre-stored log template.

[0096] Optionally, a preset log template memory is obtained; a pre-stored log template is obtained based on the preset log template memory; and the log information group is matched based on the pre-stored log template.

[0097] Optionally, the log information group is matched according to the pre-stored log template, including: sorting the pre-stored log templates according to their log lengths and marking the log lengths; obtaining the log lengths of the log information in the log information group; traversing the log information in the log information group; determining the pre-stored log templates to be matched based on the log lengths of the log information and the log lengths of the pre-stored log templates; and performing matching processing on the log information from the pre-stored log templates to be matched.

[0098] Optionally, the pre-stored log template to be matched is determined based on the log length of the log information and the log length of the pre-stored log template, including: determining the pre-stored log template whose log length is less than or equal to the log length of the log information as the pre-stored log template to be matched.

[0099] Filtering pre-stored log templates for matching by log length can improve matching efficiency.

[0100] S205: If the match is successful, determine the target log template based on the matching result.

[0101] S206: If the match fails, the log information group is parsed based on the large language model to obtain the target log template.

[0102] Optionally, one or more unmatched log messages that failed to match are filtered out from the log message group; the unmatched log messages are input into the large language model for parsing and processing to obtain the corresponding log template; and the log template is saved to the preset log template memory.

[0103] Optionally, after step S206, the method further includes: optimizing the target log template through a self-reflection mechanism; and storing the optimized target log template.

[0104] Optimizing the target log template through a self-reflection mechanism can improve its applicability, reduce the number of queries to the large language model during log parsing, and improve the efficiency and accuracy of log parsing.

[0105] S207: Based on the target log template, compress the log information group to obtain the corresponding compressed log information.

[0106] Optionally, variable data of log information in the log information group can be extracted based on the target log template; log template-variable pairs can be generated based on the target log template and variable data to obtain the corresponding compressed log information.

[0107] Optionally, multiple log messages to be compressed in a log message group can correspond to one or more target log templates.

[0108] Optionally, after step S207, the method further includes: saving the compressed log information to a preset columnar storage database; obtaining the query keyword; querying the preset columnar storage database based on the query keyword to obtain the corresponding compressed log information; and displaying the corresponding compressed log information so that maintenance personnel can judge the system's operating status and health status based on the compressed log information.

[0109] By providing retrieval and query services through a pre-defined columnar storage database, the efficiency of querying compressed log information is improved, making it easier for maintenance personnel to judge the system's operating status and health condition.

[0110] The log information compression method provided in this application compresses log information by grouping it, then using a large language model to parse the log information groups to obtain log templates, and improving the efficiency of log template extraction by pre-storing templates. This enables efficient and accurate parsing of log data, thereby improving the error rate and efficiency of log information compression.

[0111] In one embodiment of this application, based on the above embodiments, a process of updating the pre-stored log template is further included, as detailed below: In response to the user's log template update operation, the pre-stored log template is updated.

[0112] The log information compression method provided in this application can ensure that the pre-stored log template is consistent with the latest user needs, reduce the error rate, and improve the user experience.

[0113] Figure 3 This is a flowchart illustrating another log information compression method provided in this application embodiment. The execution entity of this embodiment can be... Figure 1 The computer equipment shown is not specifically limited in this embodiment. Figure 3 As shown, the method includes:

[0114] S301: Obtain the log information to be compressed.

[0115] S302: Perform data preprocessing on the log information to be compressed to obtain preprocessed log information.

[0116] Optionally, the log component type of the log information to be compressed is obtained; the corresponding preprocessing template is matched from the preset preprocessing template library according to the log component type; and the log information to be compressed is preprocessed according to the preprocessing template to obtain the preprocessed log information.

[0117] Optionally, data preprocessing is performed on the log information to be compressed according to the preprocessing template to obtain preprocessed log information, including: matching the log information to be compressed according to the preprocessing template to obtain the field information of the log information to be compressed; and replacing the field information to obtain the preprocessed log information.

[0118] Optionally, the field information may include: log content and one or more of the following: month, date, time, process name, process identifier, log level, and component name, etc.

[0119] Accordingly, the field information is replaced to obtain preprocessed log information, including: replacing fixed fields in the field information with preset wildcards to obtain preprocessed log information.

[0120] Optionally, fixed fields may include timestamps, numeric / numerical values, and dynamic identifiers.

[0121] Optionally, the dynamic identifier can be a universally unique identifier or an Internet Protocol address, etc.

[0122] By replacing field information, the efficiency of grouping can be improved, while reducing redundant calculations in the grouping stage.

[0123] S303: Group the preprocessed log information to obtain at least one log information group.

[0124] S304: Retrieve the pre-stored log template in memory.

[0125] Optionally, in the preset log template memory, the log template and the original log from which the log template was generated are stored in a one-to-one correspondence.

[0126] S305: Match log information groups according to pre-stored log templates.

[0127] S306: Determine if the match was successful.

[0128] S307: If the match is successful, determine the target log template and the corresponding original log based on the matching result.

[0129] S308: If the matching fails, the log information group is parsed based on the large language model to generate the corresponding log template; the corresponding log template is saved to the preset log template memory, and step S304 is repeated.

[0130] Optionally, the log information group and the corresponding log template can be saved to the preset log template memory.

[0131] S309: Based on the target log template, extract the variable data of the log information group to obtain the corresponding compressed log information.

[0132] S310: Save compressed log information to a preset columnar storage database.

[0133] The log information compression method provided in this application improves the stability and accuracy of dataset migration by determining whether the migration was successful based on the original dataset and the migrated dataset.

[0134] Figure 4 This is a schematic diagram of the structure of the log information compression device provided in the embodiments of this application, as shown below. Figure 4 As shown, the log information compression device 40 provided in this embodiment includes: a first acquisition module 401, a first processing module 402, a second acquisition module 403, a second processing module 404, a determination module 405, a third processing module 406, and a fourth processing module 407.

[0135] The first acquisition module 401 is used to acquire the log information to be compressed;

[0136] The first processing module 402 is used to group the log information to be compressed to obtain at least one log information group; each log information group includes multiple log information to be compressed.

[0137] The second acquisition module 403 is used to acquire a pre-stored log template; wherein the pre-stored log template is generated by the large language model parsing multiple log information sample groups;

[0138] The second processing module 404 is used to perform matching processing on the log information group according to the pre-stored log template;

[0139] Module 405 is used to determine the target log template based on the matching result if a match is successful.

[0140] The third processing module 406 is used to parse the log information group based on the large language model to obtain the target log template if the matching fails.

[0141] The fourth processing module 407 is used to compress the log information group according to the target log template to obtain the corresponding compressed log information.

[0142] In one possible implementation, the first processing module 402 is specifically used to: determine the corresponding grouping information based on the log information to be compressed; wherein the grouping information includes at least one of log length, K prefix tag and tag string; and perform grouping processing on the log information to be compressed based on the grouping information to obtain at least one log information group.

[0143] In one possible implementation, the first processing module 402, when grouping the log information to be compressed according to the grouping information to obtain at least one log information group, specifically performs the following: performs a first-level grouping process on the log information to be compressed according to the log length to obtain a first grouping result; performs a second-level grouping process on the first grouping result according to the K-prefix tag to obtain a second grouping result; calculates the tag string similarity of the second grouping result according to the tag string; and performs a third-level grouping process on the second grouping result according to the tag string similarity to obtain at least one log information group.

[0144] In one possible implementation, the first acquisition module 401 is specifically used for: acquiring multiple log information sample groups; wherein the log information sample groups include multiple log information samples; filtering the log information sample groups to obtain multiple key log information samples; inputting the multiple key log information samples into a large language model so that the large language model outputs an initial log template based on preset prompt word examples; optimizing the initial log template through a self-reflection mechanism to obtain a log template; and storing the log template.

[0145] In one possible implementation, the first acquisition module 401, when filtering the log information sample group to obtain multiple key log information samples, specifically uses a retrieval enhancement generation algorithm based on Tanimoto similarity to iteratively filter the log information samples in the log information sample group to obtain multiple key log information samples.

[0146] In one possible implementation, the first acquisition module 401, when optimizing the initial log template through a self-reflection mechanism to obtain the log template, specifically performs the following: matching all log information samples in the log information sample group according to the initial log template; if the initial log template can match all log information samples, then the initial log template is determined as the log template; if there is any log information sample that does not match the initial log template, then a new log template is generated again using the large language model and iteratively optimized until a log template that can match all log information samples is obtained.

[0147] In one possible implementation, the log information compression device further includes:

[0148] The storage module is used to optimize the target log template through a self-reflection mechanism; and to store the optimized target log template.

[0149] In one possible implementation, the log information compression device further includes:

[0150] The update module is used to update the pre-stored log template in response to the user's log template update operation.

[0151] The log information compression device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0152] Figure 5 This is a schematic diagram of the structure of a log information compression device provided in an embodiment of this application. Figure 5 As shown, the log information compression device provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0153] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0154] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0155] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0156] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0157] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0158] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0159] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0160] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0161] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0162] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0165] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0167] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and alterations may be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of compressing log information, characterized by, The method comprises the following steps: acquiring log information to be compressed; grouping the log information to be compressed to obtain at least one log information group, each of which comprises a plurality of log information to be compressed; acquiring a pre-stored log template, wherein the pre-stored log template is generated by a large language model based on analysis of a plurality of log information sample groups; matching the log information group based on the pre-stored log template; if the matching is successful, determining a target log template based on the matching result; if the matching is unsuccessful, analyzing the log information group based on a large language model to obtain a target log template; compressing the log information group based on the target log template to obtain corresponding compressed log information.

2. The method of claim 1, wherein, The grouping of the log information to be compressed to obtain at least one log information group comprises: determining corresponding grouping information based on the log information to be compressed, wherein the grouping information comprises at least one of log length, K prefix mark and mark string; grouping the log information to be compressed based on the grouping information to obtain at least one log information group.

3. The method of claim 2, wherein, The grouping of the log information to be compressed based on the grouping information to obtain at least one log information group comprises: first-level grouping of the log information to be compressed based on the log length to obtain a first grouping result; second-level grouping of the first grouping result based on the K prefix mark to obtain a second grouping result; calculating the mark string similarity of the second grouping result based on the mark string; third-level grouping of the second grouping result based on the mark string similarity to obtain at least one log information group.

4. The method according to any one of claims 1 to 3, characterized in that, Before acquiring the pre-stored log template, the method further comprises: acquiring a plurality of log information sample groups, wherein the log information sample groups comprise a plurality of log information samples; screening the log information sample groups to obtain a plurality of key log information samples; inputting the plurality of key log information samples into a large language model to enable the large language model to output an initial log template based on a pre-set prompt word example; optimizing the initial log template based on a self-reflection mechanism to obtain a log template; storing the log template.

5. The method of claim 4, wherein, The screening of the log information sample groups to obtain a plurality of key log information samples comprises: iterative screening of the log information samples in the log information sample groups based on a retrieval enhancement generation algorithm based on Tanimoto similarity to obtain a plurality of key log information samples.

6. The method of claim 5, wherein, The optimization of the initial log template based on a self-reflection mechanism to obtain a log template comprises: matching all log information samples in the log information sample groups based on the initial log template; if the initial log template can match all log information samples, determining the initial log template as a log template; If there is any log information sample that does not match the initial log template, a large language model is re-employed to generate a new log template and iterative optimization is performed until a log template that can match all the log information samples is obtained.

7. The method according to any one of claims 1 to 3, characterized in that, After the log information group is parsed based on the large language model to obtain the target log template if the matching is unsuccessful, the method further includes: optimizing the target log template through a self-reflection mechanism; storing the target log template after the optimization.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: updating the pre-stored log template in response to a log template update operation of a user.

9. An apparatus for compressing log information, characterized by comprising: The method includes: a first acquisition module configured to acquire log information to be compressed; a first processing module configured to group process the log information to be compressed to obtain at least one log information group; each log information group includes a plurality of log information to be compressed; a second acquisition module configured to acquire a pre-stored log template; the pre-stored log template is generated by parsing a plurality of log information sample groups by a large language model; a second processing module configured to match process the log information group according to the pre-stored log template; a determination module configured to, if the matching is successful, determine a target log template according to a matching result; a third processing module configured to, if the matching is unsuccessful, parse the log information group based on a large language model to obtain a target log template; a fourth processing module configured to compress process the log information group according to the target log template to obtain corresponding compressed log information.

10. A log information compression device characterized by comprising: The method includes: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method according to any one of claims 1-8.

12. A computer program product, characterised in that, The computer program is executed by the processor to implement the method according to any one of claims 1-8.

Images