Abnormal log identification method and device
By clustering and filtering representative logs from log data, and combining analysis with multiple large language models, the problems of low efficiency and high resource consumption in log anomaly identification have been solved, achieving efficient and accurate log identification and rapid repair suggestions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-03-20
AI Technical Summary
Existing log anomaly identification methods suffer from low identification efficiency, poor real-time performance, and high computational resource consumption.
By clustering the formatted log data, representative logs are selected, and semantic analysis is performed using multiple large language models. The final log recognition result is determined by combining the majority rule or the principle with the highest confidence weight.
It improves the processing efficiency and real-time performance of large language models, reduces the demand for computing resources, enhances the accuracy and reliability of log recognition results, and provides suggestions for repairing abnormal logs.
Smart Images
Figure CN121706785A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of operation and maintenance, and particularly relates to an abnormal log identification method and device. BACKGROUND
[0002] At present, in an IT operation and maintenance system, logs are widely used in fault diagnosis, performance monitoring and security audit, as important feedback information of system running state. Traditional log analysis mainly includes the following two categories: one is to match abnormal keywords (such as: ERROR) corresponding to the scene from the logs to identify abnormal logs by relying on manual experience or keyword matching based on rules; the other is to analyze the semantics of the logs by a large language model to identify abnormal logs. However, the existing log abnormal identification methods mostly have problems of low identification efficiency, poor real-time performance and large consumption of computing resources. SUMMARY
[0003] The present application provides an abnormal log identification method and device to solve the above technical problems in the prior art.
[0004] The present application provides an abnormal log identification method, comprising the following steps: performing format processing on the obtained original log data to obtain formatted log data; performing clustering on the formatted log data to obtain a plurality of log clusters; selecting a representative log from each of the log clusters; inputting a prompt word and each of the representative logs into a large language model to obtain log identification results corresponding to each of the log clusters output by the large language model, wherein the prompt word is used to instruct the large language model to perform semantic analysis on each of the representative logs to output the log identification results.
[0005] According to the abnormal log identification method provided by the present application, the formatted log data is clustered to obtain a plurality of log clusters, comprising: performing clustering on the formatted log data by taking log format as a clustering feature or taking text content of a log content field in the log data as a clustering feature to obtain a plurality of log clusters; selecting a representative log from each of the log clusters, comprising: for each of the log clusters, classifying the log content field in all log data in the cluster, and the log data with the same text content of the log content field is a log class; selecting any log data in the log class with the largest number of logs as the representative log.
[0006] The abnormal log identification method provided by the application comprises the following steps: clustering formatted log data by taking log format as a clustering feature to obtain a plurality of log clusters, wherein the method comprises the following steps: Step 1: selecting a plurality of formatted log data as clustering centers; Step 2: calculating the distance between any formatted log data and the plurality of clustering centers respectively, and classifying any formatted log data and the clustering center with the minimum distance as a cluster, wherein the distance is used to represent the similarity between any formatted log data and the corresponding clustering center; Step 3: for any cluster, taking the average value of the field values corresponding to all formatted log data in the cluster to determine a new clustering center with each average value; Step 4: repeating steps 2 and 3 until the clustering centers of the clusters no longer change or no formatted log data is re-assigned to different clusters, and finally forming a plurality of log clusters.
[0007] The abnormal log identification method provided by the application comprises the following steps: clustering formatted log data by taking log format as a clustering feature to obtain a plurality of log clusters, wherein the method comprises the following steps: inputting the prompt word and each representative log into a plurality of large language models respectively to obtain a plurality of reference identification results corresponding to each log cluster output by the plurality of large language models respectively; determining the reference identification result with the largest number as the log identification result.
[0008] The abnormal log identification method provided by the application comprises the following steps: clustering formatted log data by taking log format as a clustering feature to obtain a plurality of log clusters, wherein the method comprises the following steps: inputting the prompt word and each representative log into a plurality of large language models respectively to obtain a plurality of reference identification results corresponding to each log cluster output by the plurality of large language models respectively, wherein each large language model is provided with different confidence weights; determining the reference identification result output by the large language model with the largest confidence weight as the log identification result.
[0009] The abnormal log identification method provided by the application comprises the following steps: clustering formatted log data by taking log format as a clustering feature to obtain a plurality of log clusters, wherein the method comprises the following steps:
[0010] The abnormal log identification method provided by the application comprises the following steps: The repair suggestion is input into the large language model with the largest confidence weight, so that the large language model with the largest confidence weight outputs after integrating each repair suggestion.
[0011] The application further provides an abnormal log identification device comprising the following modules: The log formatting module is configured to perform formatting processing on the obtained original log data to obtain formatted log data. The log clustering module is configured to cluster the formatted log data to obtain a plurality of log clusters. The log screening module is configured to screen representative logs from each log cluster. The log identification module is configured to input prompt words and each representative log into a large language model to obtain log identification results output by the large language model corresponding to each log cluster, wherein the prompt words are used to instruct the large language model to perform semantic analysis on each representative log to output the log identification results.
[0012] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the abnormal log identification method of any one of the above-mentioned methods when executing the program.
[0013] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the abnormal log identification method of any one of the above-mentioned methods.
[0014] The abnormal log identification method and device provided by the application cluster the formatted log data to obtain a plurality of log clusters, only select one representative log in different log clusters to input into a large language model for semantic analysis, greatly reduce the amount of log data input into the large language model for processing, improve the processing efficiency and real-time performance of the large language model, reduce the need for computing resources of the large language model, and save computing resources. When clustering log data, the formatted log data is clustered by using the above-mentioned log format as a clustering feature, the features of each field are comprehensively considered during clustering, so that more accurate clustering results can be obtained, and the large language model can obtain accurate log identification results. Moreover, a plurality of large language models are used for abnormal log identification, the plurality of large language models can verify each other, and the log identification result is finally output by using the principle of minority submission to majority or the principle of outputting the large language model with the highest confidence weight, thereby further improving the accuracy and reliability of the log identification result. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0016] Figure 1 This is one of the flowcharts of the abnormal log identification method provided by the present invention.
[0017] Figure 2 This is a schematic diagram illustrating the principle of the abnormal log identification method provided by the present invention.
[0018] Figure 3 This is the second flowchart of the abnormal log identification method provided by the present invention.
[0019] Figure 4 This is the third flowchart of the abnormal log identification method provided by the present invention.
[0020] Figure 5 This is the fourth flowchart of the abnormal log identification method provided by the present invention.
[0021] Figure 6 This is the fifth flowchart of the abnormal log identification method provided by the present invention.
[0022] Figure 7 This is a schematic diagram of the abnormal log identification device provided by the present invention.
[0023] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] The abnormal log identification method of this invention, as shown in the embodiments, Figure 1 and 2 As shown, the procedure includes steps S110 to S140.
[0026] Step S110: Format the obtained raw log data to obtain formatted log data.
[0027] Raw log data is collected from IT, network, or other systems. It can be collected using log collection tools such as Filebeat, Logstash, and Kafka. The collected raw log data is formatted using various formatting engines, such as single-line log processing engines, multi-line log processing engines, single-line regular expression processing engines, multi-line regular expression processing engines, JSON type log processing engines, and delimiter log processing engines. The purpose of formatting is to transform the raw log data into log data in a predetermined format. For example, formatted log data may have the following field format: [Time] [Service Module] [Operation Type] [Status Code] [Response Time] [User ID] [Log Content].
[0028] It is understandable that the formatting process includes a preprocessing step for the raw log data. This preprocessing includes deduplication and desensitization of the raw log data. For example, strings containing sensitive information such as user ID numbers, phone numbers, and passwords in the logs are replaced with "****" to remove sensitive information and prevent the leakage of user privacy.
[0029] Step S120: Cluster the formatted log data to obtain multiple log clusters. In a large amount of formatted log data, log data under normal system operation often exhibits patterns (e.g., user login → query → logout, stable response time). However, log data under different abnormal system states also shows different patterns, such as consecutive 500 status codes and / or sudden spikes in response time. In this step, an unsupervised clustering algorithm can be used to cluster the formatted logs. Unsupervised clustering algorithms can group similar log data into a single group based on the characteristics of the formatted log data itself, thus obtaining multiple log clusters. For example, in the simplest two-class case, log data under normal system operation is clustered into one log cluster, while log data that does not conform to the characteristics of log data under normal system operation is naturally isolated and clustered into another log cluster.
[0030] It is understandable that all log data in each log cluster has similarity, and these log data can be considered as logs with similar semantics.
[0031] Specifically, in order to cluster the formatted log data more accurately, the distance between the two cluster centers and the formatted log data is calculated using either the log format or the text content of the log content field as the cluster feature during clustering.
[0032] For example, the formatted log format is: [Time] [Service Module] [Operation Type] [Status Code] [Response Time] [User ID] [Log Content]. When the system is in an abnormal state, some fields in the format may be missing, i.e., the value is empty, or the field value is abnormally large or small.
[0033] For cases where log format is used as a clustering feature, such as Figure 3 As shown, the clustering step in step S120 includes: Step S121: Select multiple formatted log data as cluster centers. This can be done by randomly selecting multiple formatted log data as cluster centers.
[0034] Step S122: Calculate the distance between any formatted log data and multiple cluster centers, and group any formatted log data into the cluster with the smallest distance to it. The distance is used to characterize the similarity between any formatted log data and its corresponding cluster center; that is, the smaller the distance, the more similar the formatted log data is to its corresponding cluster center, and the greater the probability that they belong to the same cluster.
[0035] For example, the distance can be obtained by calculating the Euclidean distance between the corresponding field values, as shown in the following formula: .
[0036] in, X This represents any formatted log data. Y Indicates the cluster center. x i This represents any formatted log data. i The field value of the field. y i Indicates the cluster center number i The field value of the field. n This represents the total number of fields. It should be noted that for fields with numeric values, To represent the difference in numerical values, in the case of strings, This represents the similarity of strings converted into numerical vectors. For example, methods such as TF-IDF, word embeddings, or BERT can be used to convert strings into numerical vectors.
[0037] Step S123: For any cluster, take the average of the field values corresponding to all formatted log data in it, and use the average values to determine the new cluster center.
[0038] Step S124: Repeat steps S122 and S123 until the cluster centers of each cluster no longer change or there is no formatted log data redistributed to different clusters, and finally multiple clusters are formed into multiple log clusters.
[0039] In this embodiment, the clustering process that uses the above-mentioned log format as a clustering feature is used to cluster the formatted log data. The characteristics of each field are fully considered during the clustering process, so as to obtain more accurate clustering results. That is, the type of formatted log data in each log cluster is basically the same, either all are normal logs or all are abnormal logs of the same type (e.g., abnormal response time).
[0040] For cases where the text content of the log content field in the log data is used as the clustering feature, the clustering steps are basically the same as steps one through four above. The difference is that only the log content field is used to calculate the distance. Since the log content field contains the richest semantic information, for example: "[2025-09-18 10:00:00] [Order Service][Payment][ERROR: 500][Time taken: 1500ms] [user123][Order payment interface call failed]", the specific value of the log content field is: "Order payment interface call failed". This reflects that the service module is for order services, the operation type is payment, and since the final payment failed, it implies that the payment process will continuously retry, resulting in a long processing time. Therefore, in the clustering process, using only the text content of the log content field in the log data as the clustering feature not only ensures the relative accuracy of the clustering results but also reduces the computational load and improves the efficiency of clustering.
[0041] Step S130: Select representative logs from each log cluster. Since all log data in the same log cluster has semantic similarity, all log data within the cluster are either normal logs or abnormal logs of the same type. Therefore, a representative log can be selected from each log cluster for subsequent semantic analysis using a large language model to obtain the log recognition result. This log recognition result is the log recognition result for all log data within the cluster.
[0042] For example, representative logs can be randomly selected from log clusters or selected according to pre-defined rules. For instance, the log data with the longest text length in the log content field can be selected. The longer the text length, the richer the semantic information contained, and the more accurate the log recognition results can be obtained.
[0043] Specifically, such as Figure 4 As shown, step S130 includes: Step S131: For each log cluster, classify the log data within the cluster according to the log content field. Log data with the same text content corresponding to the log content field belong to one log class. That is, all structured log data in the same log cluster are divided into several subclasses according to the different values of the log content field.
[0044] Step S132: Select any log data from the log class with the largest number of logs as the representative log, that is, select the log data with the highest frequency of log content in the log cluster as the representative log.
[0045] In this embodiment, since the log content is more indicative of the type of log data, the log data with the highest frequency of log content in the log cluster is selected as the representative log, so that the subsequent large language model can output more accurate log recognition results.
[0046] Step S140: Input the prompt words and each of the representative logs into the large language model to obtain the log recognition results for each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs to output the log recognition results. It is understood that the log recognition results include two types: normal logs and abnormal logs.
[0047] For example, the prompt words of the large language model are shown in Table 1 below.
[0048] Table 1. One of the prompt words in the large language model
[0049] The large language model interprets log content based on prompt words, determines whether it belongs to abnormal logs or normal logs, and outputs the result.
[0050] To obtain more accurate log identification results, such as Figure 5 As shown, one implementation of step S140 includes the following steps: Step S141a: Input the prompt words and each representative log into multiple large language models, respectively, to obtain multiple reference recognition results corresponding to each log cluster output by the multiple large language models. For example, input the prompt words and each representative log into three large language models (e.g., DeepSeek, Qwen, and BERT), and each large language model will output its own reference recognition result.
[0051] Step S142a: The log identification result with the largest number of reference identification results is determined as the log identification result, that is, the final log identification result is determined according to the principle of majority rule.
[0052] The log recognition results output by multiple large language models are determined by majority rule. The log recognition results of multiple large language models are mutually verified to avoid the problem of inaccurate log recognition results caused by inference errors of a single large language model, thereby improving the credibility of the log recognition results.
[0053] To obtain more accurate log identification results, such as Figure 6 As shown, another implementation of step S140 includes the following steps: Step S141b: Input the prompt words and each representative log into multiple large language models, respectively, to obtain multiple reference recognition results for each log cluster output by the multiple large language models. Each large language model has a different confidence weight, which represents the reliability of the output result. The confidence weight can be set based on the accuracy of the log recognition results historically output by the large language model; the higher the accuracy, the greater the confidence weight. The accuracy of the log recognition results historically output by the large language model can be verified and statistically obtained by maintenance personnel.
[0054] Step S142b: Determine the reference recognition result output by the large language model with the highest confidence weight as the log recognition result.
[0055] By using the output of the large language model with the highest confidence weight as the standard, the accuracy and reliability of the log recognition results output by the large language model are improved.
[0056] The abnormal log identification method of this embodiment of the invention significantly reduces the amount of log data input to the large language model for semantic analysis by clustering the formatted log data into multiple log clusters and selecting only a representative log from each cluster. This improves the processing efficiency and real-time performance of the large language model, reduces its computational resource requirements, and saves computing resources. When clustering the log data, the clustering process using the aforementioned log format as a clustering feature comprehensively considers the characteristics of each field, resulting in more accurate clustering results and enabling the large language model to obtain accurate log identification results. Furthermore, by using multiple large language models for abnormal log identification, these models can mutually verify each other. The final log identification result is output based on the principle of majority rule or the principle of using the output of the large language model with the highest confidence weight, further improving the accuracy and reliability of the log identification results.
[0057] In some embodiments, the prompt word is also used to instruct the large language model to output a repair suggestion for the abnormal log (e.g., restart the service, update the configuration, check the network connection, etc.) based on a preset historical knowledge base when the output log recognition result is an abnormal log. The historical knowledge base stores a mapping table of abnormal logs and corresponding repair suggestions.
[0058] Specifically, the prompt words, historical knowledge base, and representative logs are all input into the large language model. The large language model performs semantic analysis on the representative logs based on the prompt words and outputs the log recognition result, i.e., outputs normal logs or abnormal logs. In the case of outputting abnormal logs, the model combines the mapping table of abnormal logs and corresponding repair suggestions in the historical knowledge base to output the corresponding repair suggestions for the abnormal logs. For example, in this embodiment, the prompt words are shown in Table 2 below.
[0059] Table 2. Prompt words from the large language model (Part 2)
[0060] In this embodiment, the large language model is guided by prompt words to output corresponding repair suggestions in addition to outputting anomaly logs, so that operation and maintenance personnel can refer to the repair suggestions and quickly handle system anomalies.
[0061] Furthermore, the prompt also includes instructions to the large language model to extract key anomaly information from the log and output the anomaly type when the log is determined to be an anomaly. This is used to extract key anomaly information (such as anomaly type, error code, affected components, etc.) when the large language model determines that the log is an anomaly, and to output the anomaly type of the log, which helps operations and maintenance personnel understand the cause of the anomaly.
[0062] In some embodiments, when the log recognition results output by each of the large language models are the same, and after each of the large language models outputs the repair suggestions for the abnormal logs, the method further includes: Each repair suggestion is input into the large language model with the highest confidence weight, so that the large language model with the highest confidence weight integrates the repair suggestions and outputs them. That is, the large language model with the highest confidence weight summarizes all the repair suggestions and outputs them. Of course, if the log recognition results output by the various language models are different, the repair suggestion output by the large language model with the highest confidence weight shall prevail.
[0063] In this embodiment, the large language model with the highest confidence weight has better semantic understanding and analysis capabilities. The repair suggestions output by various semantic models are processed by the large language model with the highest confidence weight before being output, so that the output repair suggestions have higher confidence and operability.
[0064] Furthermore, in some embodiments, the abnormal log identification method further includes: receiving the adjustment amount of the confidence weights of various language models, which is determined based on the log identification results and / or repair suggestions output by the operations and maintenance personnel. Specifically, if the log identification results and / or repair suggestions output by the language models are verified to be correct, the adjustment amount is set to a positive value; otherwise, it is a negative value. And when the confidence weight of any language model is less than a preset weight threshold, a message indicating low confidence of the language model is displayed to remind the operations and maintenance personnel to replace the language model to avoid erroneous output, or to adjust the prompt words so that any of the language models obtains the correct log identification result.
[0065] The abnormal log identification device provided by the present invention is described below. The abnormal log identification device described below can be referred to in correspondence with the abnormal log identification method described above.
[0066] The abnormal log identification device of this invention, such as Figure 7 As shown, it includes: The log formatting module 710 is used to format the acquired raw log data to obtain formatted log data.
[0067] The log clustering module 720 is used to cluster formatted log data to obtain multiple log clusters.
[0068] The log filtering module 730 is used to filter representative logs from each of the log clusters.
[0069] The log recognition module 740 is used to input prompt words and each of the representative logs into the large language model to obtain the log recognition results corresponding to each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
[0070] The abnormal log identification device in this embodiment clusters the formatted log data to obtain multiple log clusters. Only one representative log from each log cluster is selected and input into the large language model for semantic analysis. This greatly reduces the amount of log data input into the large language model for processing, improves the processing efficiency and real-time performance of the large language model, reduces the large language model's need for computing resources, and saves computing resources.
[0071] In some embodiments, the log clustering module 720 is specifically used to cluster the formatted log data using the log format as a clustering feature or using the text content of the log content field in the log data as a clustering feature, to obtain multiple log clusters.
[0072] The log filtering module 730 specifically includes: The intra-cluster classification module is used to classify log data within each log cluster according to the log content field in all log data, wherein log data with the same text content corresponding to the log content field belong to one log class.
[0073] The representative log selection module is used to select any log data from the log class with the largest number of logs as the representative log.
[0074] In some embodiments, the log clustering module 720 is specifically used to perform the following steps 1 to 4: Step 1: Select multiple formatted log data sets as cluster centers; Step 2: Calculate the distance between any formatted log data and multiple cluster centers, and group any formatted log data into a cluster with the smallest distance to it. The distance is used to characterize the similarity between any formatted log data and the corresponding cluster center. Step 3: For any cluster, take the average of the field values corresponding to all formatted log data in it, and use the average values to determine the new cluster center; Step 4: Repeat steps 2 and 3 until the cluster centers of each cluster no longer change or there is no formatted log data being redistributed to different clusters, ultimately forming multiple log clusters.
[0075] In some embodiments, the log recognition module 740 is specifically used to input the prompt words and each of the representative logs into multiple large language models respectively, and obtain multiple reference recognition results corresponding to each log cluster output by the multiple large language models respectively; and determine the reference recognition result with the most references as the log recognition result.
[0076] In some embodiments, the log recognition module 740 is specifically used to input the prompt words and each of the representative logs into multiple large language models respectively, and obtain multiple reference recognition results corresponding to each log cluster output by the multiple large language models respectively, wherein each of the large language models has different confidence weights; the reference recognition result output by the large language model with the largest confidence weight is determined as the log recognition result.
[0077] In some embodiments, the prompt word is also used to instruct the large language model to output a repair suggestion for the abnormal log based on a preset historical knowledge base when the output log recognition result is an abnormal log. The historical knowledge base stores a mapping table of abnormal logs and corresponding repair suggestions.
[0078] In some embodiments, the log recognition module 740 is further configured to input each repair suggestion into the large language model with the highest confidence weight when the log recognition results output by each of the large language models are the same, and after each of the large language models outputs the repair suggestions for the abnormal log, so that the large language model with the highest confidence weight integrates and outputs the repair suggestions.
[0079] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an exception log identification method, which includes: The obtained raw log data is formatted to obtain formatted log data.
[0080] The formatted log data is clustered to obtain multiple log clusters.
[0081] Select representative logs from each of the log clusters.
[0082] The prompt words and each of the representative logs are input into the large language model to obtain the log recognition results for each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
[0083] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0084] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the abnormal log identification method provided by the above methods, the method comprising: The obtained raw log data is formatted to obtain formatted log data.
[0085] The formatted log data is clustered to obtain multiple log clusters.
[0086] Select representative logs from each of the log clusters.
[0087] The prompt words and each of the representative logs are input into the large language model to obtain the log recognition results for each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the anomaly log identification method provided by the above methods, the method comprising: The obtained raw log data is formatted to obtain formatted log data.
[0089] The formatted log data is clustered to obtain multiple log clusters.
[0090] Select representative logs from each of the log clusters.
[0091] The prompt words and each of the representative logs are input into the large language model to obtain the log recognition results for each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An anomaly log identification method, characterized in that, include: The obtained raw log data is formatted to obtain formatted log data; The formatted log data is clustered to obtain multiple log clusters; Select representative logs from each of the log clusters; The prompt words and each of the representative logs are input into the large language model to obtain the log recognition results for each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
2. The abnormal log identification method according to claim 1, characterized in that, The formatted log data is clustered to obtain multiple log clusters, including: The formatted log data is clustered using either the log format as a clustering feature or the text content of the log content field as a clustering feature, resulting in multiple log clusters. Select representative logs from each of the log clusters, including: For each log cluster, log data is classified according to the log content field in all log data within the cluster. Log data with the same text content corresponding to the log content field belong to one log class. Select any log data from the log class with the most logs as the representative log.
3. The abnormal log identification method according to claim 2, characterized in that, Clustering the formatted log data using log format as a clustering feature yields multiple log clusters, including: Step 1: Select multiple formatted log data sets as cluster centers; Step 2: Calculate the distance between any formatted log data and multiple cluster centers, and group any formatted log data into a cluster with the smallest distance to it. The distance is used to characterize the similarity between any formatted log data and the corresponding cluster center. Step 3: For any cluster, take the average of the field values corresponding to all formatted log data in it, and use the average values to determine the new cluster center; Step 4: Repeat steps 2 and 3 until the cluster centers of each cluster no longer change or there is no formatted log data being redistributed to different clusters, ultimately forming multiple log clusters.
4. The abnormal log identification method according to claim 1, characterized in that, The prompt words and each representative log entry are input into the large language model to obtain the log recognition results for each log cluster output by the large language model, including: The prompt words and each of the representative logs are input into multiple large language models, and multiple reference recognition results corresponding to each log cluster are obtained from the output of the multiple large language models. The log identification result with the largest number of reference identification results is determined as the log identification result.
5. The abnormal log identification method according to claim 1, characterized in that, The prompt words and each representative log entry are input into the large language model to obtain the log recognition results for each log cluster output by the large language model, including: The prompt words and each of the representative logs are respectively input into multiple large language models, and multiple reference recognition results corresponding to each log cluster are obtained from the output of the multiple large language models. Each of the large language models has different confidence weights. The reference recognition result output by the large language model with the highest confidence weight is determined as the log recognition result.
6. The abnormal log identification method according to claim 5, characterized in that, The prompt words are also used to instruct the large language model to output repair suggestions for the abnormal log based on a preset historical knowledge base when the log recognition result is an abnormal log. The historical knowledge base stores a mapping table of abnormal logs and corresponding repair suggestions.
7. The abnormal log identification method according to claim 6, characterized in that, If the log recognition results output by each of the large language models are the same, and after each of the large language models outputs the repair suggestions for the abnormal logs, the method further includes: Each repair suggestion is input into the large language model with the highest confidence weight, so that the large language model with the highest confidence weight can integrate and output the repair suggestions.
8. An anomaly log identification device, characterized in that, include: The log formatting module is used to format the acquired raw log data to obtain formatted log data; The log clustering module is used to cluster formatted log data to obtain multiple log clusters; A log filtering module is used to filter representative logs from each of the log clusters; The log recognition module is used to input prompt words and each of the representative logs into the large language model to obtain the log recognition results of each log cluster output by the large language model. The prompt words are used to instruct the large language model to perform semantic analysis on each of the representative logs in order to output the log recognition results.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the abnormal log identification method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal log identification method as described in any one of claims 1 to 7.