Fault processing method and device based on log analysis, equipment, medium and product

Abstract

The invention relates to a fault processing method and device based on log analysis, equipment, a medium and a product. The method comprises the following steps: classifying and preprocessing logs and monitoring indexes to obtain different types of structured data; performing feature extraction and clustering on different types of structured data to obtain cue words describing log contents; according to the prompt word, the log content is marked and analyzed through a large model, and a log tag and an analysis result are obtained; constructing an operation and maintenance knowledge base based on the log tag and the analysis result; constructing a fault countermeasure knowledge base, and constructing a processing scheme agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; the processing scheme intelligent agent automatically matches the fault countermeasures according to the log content, and triggers the execution of the corresponding fault countermeasures. Therefore, the logs can be automatically screened and labeled, the labor cost is reduced, the log content and the fault countermeasures are automatically matched through the processing scheme intelligent agent associated with the knowledge base, and the fault processing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a fault handling method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on log analysis. Background Art

[0002] Log analysis and troubleshooting are key links in system operation and maintenance. With the continuous development of information technology, their complexity and diversity are also increasing. Since logs come from a wide range of sources and contain rich information, they have difficulties such as large data volume, inconsistent formats, and complex content.

[0003] In traditional technology, regular expressions are generally written to match patterns in logs to achieve analysis and troubleshooting. However, writing regular expressions requires a certain level of technical skills and experience. Complex log patterns may take a lot of time and effort to build regular expressions, and the matching results may result in false positives and false negatives, affecting the accuracy of troubleshooting. Summary of the invention

[0004] Based on this, it is necessary to provide a log analysis-based fault handling method, device, computer equipment, computer-readable storage medium and computer program product that can reduce manual intervention and improve the accuracy of fault handling in response to the above technical problems.

[0005] In a first aspect, the present application provides a fault handling method based on log analysis, comprising:

[0006] Classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0007] Perform feature extraction and clustering on different types of structured data to obtain prompt words describing the log content;

[0008] According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results;

[0009] Building an operation and maintenance knowledge base based on the log tags and the analysis results;

[0010] Constructing a fault countermeasure knowledge base, and constructing a processing solution intelligent entity based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0011] The processing scheme agent automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures.

[0012] In one embodiment, the logs and monitoring indicators are classified and preprocessed to obtain different types of structured data, including:

[0013] According to the business system and the components that the business system depends on, the logs and monitoring indicators are divided into: application logs, operation logs, component logs, and host-related monitoring indicators;

[0014] The semi-structured data in the application log, operation log, and component log are converted into structured data respectively to obtain structured data of different categories.

[0015] In one embodiment, converting the semi-structured data in the application log, the operation log, and the component log into structured data respectively includes:

[0016] Extracting information from semi-structured data in application logs, operation logs, and component logs to obtain key information, including timestamps, event types, source addresses, and target addresses in the logs.

[0017] If the application log, operation log, or component log contains user privacy data, encrypt or hide the user privacy data, and retain identification information indicating the user system;

[0018] The key information is stored in a preset field format to obtain structured data.

[0019] In one embodiment, the feature extraction and clustering of different types of structured data to obtain prompt words describing the log content includes:

[0020] Perform data cleaning on different categories of structured data to obtain cleaned data;

[0021] Create log parsing and clustering pipelines for each type of cleaned data by category;

[0022] The log is split through log parsing and clustering pipeline to obtain several strings of the same length, and the strings are stored in a distributed file system for parallel parsing to obtain the frequency of occurrence of each string;

[0023] After sorting the strings according to their frequencies of occurrence from large to small, clustering the log contents in the strings to obtain K clusters; each cluster corresponds to a log label, and K is a natural number greater than 0;

[0024] The logs marked with log tags are input into a pre-established prompt word template to generate prompt words describing the log content.

[0025] In one embodiment, the step of constructing a fault countermeasure knowledge base includes:

[0026] Building a fault countermeasure knowledge base based on historical processing experience, expert knowledge, solutions to unique services, and preset general solutions; wherein the solutions to unique services are jointly established by developers, experts, and operation and maintenance personnel;

[0027] The general solution includes: alarm templates for monitoring indicators and message notifications of business anomalies.

[0028] In one embodiment, the method further comprises:

[0029] Obtain the operating status, processing effect, and overall performance parameters of the operation and maintenance system;

[0030] According to the operation status, processing effect and overall performance parameters of the operation and maintenance system, the accuracy of the fault countermeasures is evaluated to obtain an evaluation result;

[0031] According to the evaluation result, the log content and fault countermeasures associated with the processing solution agent are adjusted.

[0032] In a second aspect, the present application also provides a fault handling device based on log analysis, comprising:

[0033] The classification and preprocessing module is used to classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0034] The extraction and clustering module is used to extract features and cluster different types of structured data to obtain prompt words describing the log content;

[0035] An analysis module is used to mark and analyze the log content through a large model according to the prompt words to obtain log tags and analysis results;

[0036] An operation and maintenance knowledge base construction module, used to construct an operation and maintenance knowledge base based on the log tags and the analysis results;

[0037] A fault countermeasure knowledge base construction module is used to construct a fault countermeasure knowledge base;

[0038] A processing solution agent building module, used to build a processing solution agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0039] The execution module is used to automatically match the fault countermeasures according to the log content through the processing solution agent, and trigger the execution of the corresponding fault countermeasures.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] Classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0042] Perform feature extraction and clustering on different types of structured data to obtain prompt words describing the log content;

[0043] According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results;

[0044] Building an operation and maintenance knowledge base based on the log tags and the analysis results;

[0045] Constructing a fault countermeasure knowledge base, and constructing a processing solution intelligent entity based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0046] The processing scheme agent automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0048] Classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0049] Perform feature extraction and clustering on different types of structured data to obtain prompt words describing the log content;

[0050] According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results;

[0051] Building an operation and maintenance knowledge base based on the log tags and the analysis results;

[0052] Constructing a fault countermeasure knowledge base, and constructing a processing solution intelligent entity based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0053] The processing scheme agent automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures.

[0054] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0055] Classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0056] Perform feature extraction and clustering on different types of structured data to obtain prompt words describing the log content;

[0057] According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results;

[0058] Building an operation and maintenance knowledge base based on the log tags and the analysis results;

[0059] Constructing a fault countermeasure knowledge base, and constructing a processing solution intelligent entity based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0060] The processing scheme agent automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures.

[0061] The above-mentioned fault handling method, device, computer equipment, computer-readable storage medium and computer program product based on log analysis obtain structured data of different categories by classifying and preprocessing logs and monitoring indicators; thereby facilitating the subsequent targeted processing of different types of logs and improving processing efficiency and accuracy. Feature extraction and clustering are performed on structured data of different categories to obtain prompt words describing the log content; thereby, the log content can be simply and quickly characterized by the prompt words, which is convenient for subsequent labeling and analysis of the log content. According to the prompt words, the log content is labeled and analyzed by a large model to obtain log labels and analysis results; an operation and maintenance knowledge base is constructed based on the log labels and the analysis results; thereby, log labels and analysis results of different categories of log content can be automatically labeled, which is convenient for subsequent rapid matching of fault countermeasures. A fault countermeasure knowledge base is constructed, and a processing solution intelligent body is constructed based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; thereby, the automatic selection and recommendation of information in the knowledge base is directly realized by the processing solution intelligent body without manual intervention. The processing solution intelligent body automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures. This allows logs to be automatically filtered and annotated, reducing the cost of manual data annotation, and building a higher-quality knowledge base that better meets the needs of the target system based on the automatically filtered and annotated log content. By associating the knowledge base's processing solution agent to automatically match log content and fault countermeasures, the efficiency and accuracy of fault handling are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 An application environment diagram of a fault handling method based on log analysis in one embodiment;

[0064] Figure 2 The figure is a flow chart of a fault handling method based on log analysis in one embodiment;

[0065] Figure 3 It is a flowchart of a fault handling method based on log analysis in another embodiment;

[0066] Figure 4 is a structural block diagram of a fault processing device based on log analysis in one embodiment;

[0067] Figure 5 is a structural block diagram of a fault processing device based on log analysis in another embodiment;

[0068] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] In order to facilitate understanding of the various technical solutions in the embodiments of the present application, the professional terms appearing in the embodiments are briefly explained as follows:

[0071] Agent: refers to an entity with the ability to make decisions and take actions autonomously. In the field of computer science and artificial intelligence, agents are usually designed as programs or systems that can perceive the environment, learn, reason, and interact with other agents.

[0072] Intelligent Operation and Maintenance (AIOps): is an innovative approach that applies artificial intelligence technology to the field of operation and maintenance. It aims to improve operation and maintenance efficiency, reduce costs, and improve system reliability and stability by using technologies such as machine learning, big data analysis, and automation. AIOps can help operation and maintenance teams better understand and manage complex IT systems. It can automatically collect and analyze large amounts of operation and maintenance data, including logs, performance indicators, events, etc., and use machine learning algorithms to identify patterns, predict failures, optimize resource allocation, etc.

[0073] Large Language Model: A language model with a large number of parameters and complex structure. These models are usually based on deep learning techniques, such as neural networks, and can understand and generate natural language by learning and training on large amounts of text data. It has achieved remarkable results in the field of natural language processing and has a wide range of applications, such as intelligent customer service, content generation, language understanding and interaction, etc.

[0074] The fault handling method based on log analysis provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the business system may include multiple hosts (for example, host 1, host 2, host 3), and each host is loaded with corresponding applications (for example, application 1, application 2, application 3). When the application in the host is executed, operation logs, application logs, component logs, and monitoring indicators will be generated. After the operation logs, application logs, component logs, and monitoring indicators are collected and pre-classified, the transmission value log clustering module is used to establish a corresponding pipeline for each log category (for example: log splitting, hierarchical sorting, clustering analysis, prompt word division). Further, according to the divided prompt words, the log content is marked and analyzed by a large model to obtain log labels and analysis results; an operation and maintenance knowledge base is constructed based on the log labels and the analysis results; a fault countermeasure knowledge base is constructed, and a processing solution intelligent body is constructed based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; the processing solution intelligent body automatically matches the fault countermeasure according to the log content, and triggers the execution of the corresponding fault countermeasure.

[0075] For example, Figure 1 As shown in the figure, the operation and maintenance knowledge base includes application log types, component log types, operation log types, and monitoring indicator types. The fault countermeasure knowledge base includes: internal API, message notification, automatic processing, and alarm threshold. The association between the operation and maintenance knowledge base and the fault countermeasure knowledge base is established through the processing solution agent. When the log content is obtained, the fault countermeasure can be automatically matched and the corresponding fault countermeasure can be triggered.

[0076] In an exemplary embodiment, Figure 2 As shown in the figure, a fault handling method based on log analysis is provided. Figure 1 The system architecture in is taken as an example to illustrate, including the following steps 201 to 206.

[0077] Step 201 , classify and preprocess the logs and monitoring indicators to obtain structured data of different categories.

[0078] In this embodiment, it is necessary to comprehensively sort out and analyze the logs and monitoring indicators from various sources. Considering the diversity of sources, these logs and monitoring indicators may include different systems, devices, applications, etc., so they need to be carefully classified according to their characteristics, data formats, generation frequencies, etc.

[0079] Exemplarily, the logs and monitoring indicators can be divided into application logs, operation logs, component logs, and host-related monitoring indicators according to the business system and the components on which the business system depends; the semi-structured data in the application logs, operation logs, and component logs are converted into structured data to obtain different categories of structured data.

[0080] It should be noted that the monitoring indicators need to be associated with the application logs so that when analyzing faults, the reference monitoring indicators can be accurately known to improve the accuracy of fault handling.

[0081] For example, in addition to classifying by business system, it is also possible to classify by specific application modules or functions under the microservice architecture. Such classification helps to process different types of logs in a targeted manner and improve processing efficiency and accuracy.

[0082] In this embodiment, the logs often contain some irregular or non-standard data formats, so before fully analyzing the data, it is necessary to convert it into a form with a clear structure and format through a series of technical means. For example, regular expressions, pattern matching, data cleaning and other methods can be used to remove irrelevant information, extract key data, and organize it into a unified structure.

[0083] Exemplarily, information is extracted from the semi-structured data in the application log, operation log, and component log to obtain key information, including: timestamp, event type, source address, and target address in the log. The key information is stored in a preset field format to obtain structured data.

[0084] In this embodiment, important information such as the timestamp, event type, source address, and target address in the log may be extracted and stored in a specific field format for subsequent analysis and processing.

[0085] Exemplarily, if the application log, operation log, or component log contains user privacy data, the user privacy data is encrypted or hidden, and identification information indicating the user system is retained.

[0086] In this embodiment, the processed raw log data may contain user privacy data. In order to prevent user data leakage, the user privacy data needs to be encrypted (of course, hiding, replacing, etc. can also be used). In order to ensure the accuracy of the data and the quality of the knowledge base, the user data should be encrypted and its unique identification in the system should be retained. For example, the user's mobile phone number, ID card, address and other information are filtered out by regular expressions, pattern matching and other methods and encrypted using encryption methods. At the same time, the user's system ID (device identification) and uuid (universally unique identification) are retained in the log to identify the user's behavior pattern in the operation log or application log.

[0087] The steps in this embodiment can realize automatic screening and labeling of logs, reducing the cost of manual data labeling. It is convenient to build a higher quality knowledge base that better meets the needs of the target system based on the log content of automatic classification, screening and labeling, so that the constructed knowledge base can be used not only in the operation and maintenance platform, but also in more private enterprise large model training.

[0088] Step 202 , extracting and clustering features of structured data of different categories to obtain prompt words describing the log content.

[0089] In this embodiment, in each category, after extracting the cleaned log content in all categories, clustering and analysis of the log content are performed, and finally prompt words are generated. Prompt words are the basis for automatic log annotation of large models, which can be used to improve the accuracy of log marking.

[0090] Exemplarily, data cleaning is performed on structured data of different categories to obtain cleaned data; a log parsing and clustering pipeline is created for each category of cleaned data according to the category; the log is split through the log parsing and clustering pipeline to obtain several character strings of the same length, and the character strings are stored in a distributed file system for parallel parsing to obtain the frequency of occurrence of each character string; after sorting the character strings from large to small according to the frequency of occurrence, the log contents in the character strings are clustered to obtain K clusters; wherein each cluster corresponds to a log tag, and K is a natural number greater than 0; the log marked with the log tag is input into a pre-established prompt word template to generate prompt words describing the log content.

[0091] In this embodiment, due to the huge number of logs, the logs can be split into tokens (a string) of the same length and put into HDFS (distributed file system) for parallel parsing. Since there are many similar contents in the logs, such as timestamps, error types, network information, etc., the tokens can be divided according to the frequency of occurrence and sorted in order from large to small. Then, the similar log contents in the tokens are calculated and clustered, where the similarity calculation formula is as follows:

[0092]

[0093] Among them, logA represents the content of log A, logB represents the content of log B, T(logA) represents the content description of log A, and T(logB) represents the content description of log B.

[0094] By calculating the distance of similarity in the token, parallel computing and processing are performed in HDFS, and the log content is finally clustered into K clusters. Each cluster corresponds to a category, which is used to label the corresponding log content.

[0095] In this embodiment, the division and generation of prompt words generally require manual parameter adjustment and empirical analysis to create a prompt word template. For example, the formulation rules of the prompt word template include: basic requirements, task input / output, and input-to-output mapping mechanism.

[0096] It should be noted that, considering the diversity of log clustering, the input and output parameters and mapping methods need to adopt more advanced and detailed customized descriptions, such as using regular expressions to extract log content.

[0097] Exemplarily, a prompt word template may be used to input and train the above clustering, so as to obtain prompt words that describe the content of the log. It should be noted that if the prompt word is used to analyze an unlabeled (original log), the analysis result of the unlabeled log may be iterated into the above steps to optimize the prompt word template.

[0098] Exemplarily, the prompt word template may include: instruction, input, and output. The details are as follows:

[0099] instruction:

[0100] According to the parameter information (formatted parameters) and log message (log content), determine the classification of the log (for example: business error, component error, user behavior abnormality alarm, machine resource abnormality or other abnormalities).

[0101] enter:

[0102] Parameter information: {parameter information #1}

[0103] Log message: {Log message #1}

[0104] Output:

[0105] Log category: {Log category #1}

[0106] Among them: {Parameter information #1} refers to the common parameters extracted from the clustering content after log clustering, such as common machine IP, log level, context ID, instance name, instance ID, etc.; or common parameters in a certain type of business, such as user ID and order ID in the order business system; or CPU usage percentage and memory usage percentage in the system resource monitoring system. {Log message #1} refers to the original log text. {Log category #1} refers to one of the business errors, component errors, abnormal user behavior alarms, machine resource anomalies, or other anomalies declared in the "instructions". These errors can be expanded and subdivided according to actual conditions.

[0107] Step 203: According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results.

[0108] In this embodiment, these formatted prompt words and log text can be used as input, and the large model can be used to analyze the behavior, action and meaning of the log in the system, and the tags and analysis formatted into the operation and maintenance knowledge base.

[0109] It should be noted that the prompt word template in the above steps can put the unformatted original log into the large model for analysis and formatting. The formatted result is similar to a regular expression, and the prompt words can be used to parse and classify the input unformatted log.

[0110] In this embodiment, in order to ensure the accuracy of the large model, the generated prompt word label can be used to label all logs in HDFS and the labeling results can be checked. For logs that cannot be labeled or are incorrectly labeled, re-aggregation and iteration are performed to generate a more accurate prompt word template.

[0111] Step 204: construct an operation and maintenance knowledge base based on the log tags and analysis results.

[0112] In this embodiment, an association relationship between log tags and analysis results is established in the operation and maintenance knowledge base, so that the subsequent processing solution agent can quickly determine the corresponding analysis results based on the log content.

[0113] Step 205: construct a fault countermeasure knowledge base, and construct a processing solution agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base.

[0114] Exemplarily, a fault countermeasure knowledge base can be constructed based on historical processing experience, expert knowledge, business-specific solutions, and preset general solutions; wherein the business-specific solutions are jointly established by developers, experts, and operation and maintenance personnel; the general solutions include: alarm templates for monitoring indicators and message notifications of business anomalies.

[0115] In this embodiment, the processing solutions and general solutions unique to the business in the business system can be integrated and associated with the corresponding knowledge base content. The construction of the fault countermeasures knowledge base should include a wealth of processing strategies to deal with different types of log situations. Exemplarily, this knowledge base can be constructed by summarizing historical processing experience, introducing expert knowledge, etc. For example, general solutions include alarm templates for monitoring indicators, message notifications for business anomalies, etc. These general alarm methods are organized as interfaces so that the processing solution agent can learn and call them to realize the function of automatic processing.

[0116] In this embodiment, for business-specific solutions, developers, experts, and operation and maintenance personnel are required to jointly build this processing method and process it in different ways in the knowledge base, such as: internal API, internal message notification, startup fuse and current limiting, etc. These solutions need to be constructed specifically for different business systems.

[0117] The processing solution agent designed in this embodiment can automatically select and recommend appropriate processing solutions based on the information in the log analysis knowledge base. Since this processing solution agent is trained based on a large model, the quality of the knowledge base is the main factor affecting the agent's reasoning performance. That is, before the agent is trained, the quality and quantity of the logs in the knowledge base need to be filtered and verified. This enables the agent to have the reasoning ability to analyze existing logs and understand the business status of the current system based on the current log situation.

[0118] Step 206, the processing solution agent automatically matches the fault countermeasure according to the log content, and triggers the execution of the corresponding fault countermeasure.

[0119] In this embodiment, the processing solution agent has the ability to make independent decisions. After obtaining the log tag corresponding to the log content, it can quickly match the fault countermeasures and trigger the execution of the corresponding fault countermeasures to achieve automatic processing.

[0120] Exemplarily, the fault countermeasure knowledge base may provide interfaces for different countermeasures so that the processing solution agent can flexibly make interface calls.

[0121] The above-mentioned fault handling method based on log analysis obtains structured data of different categories by classifying and preprocessing logs and monitoring indicators; thus, it is convenient to process different types of logs in a targeted manner in the future, thereby improving processing efficiency and accuracy. Feature extraction and clustering are performed on structured data of different categories to obtain prompt words describing the log content; thus, the log content can be simply and quickly characterized by the prompt words, which is convenient for subsequent labeling and analysis of the log content. According to the prompt words, the log content is labeled and analyzed by a large model to obtain log labels and analysis results; an operation and maintenance knowledge base is constructed based on the log labels and the analysis results; thus, log labels and analysis results of different categories of log content can be automatically labeled, which is convenient for subsequent rapid matching of fault countermeasures. A fault countermeasure knowledge base is constructed, and a processing solution intelligent body is constructed based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; thus, no manual intervention is required, and the processing solution intelligent body directly realizes the automatic selection and recommendation of information in the knowledge base. The processing solution intelligent body automatically matches fault countermeasures according to the log content, and triggers the execution of corresponding fault countermeasures. This allows logs to be automatically filtered and annotated, reducing the cost of manual data annotation, and building a higher-quality knowledge base that better meets the needs of the target system based on the automatically filtered and annotated log content. By associating the knowledge base's processing solution agent to automatically match log content and fault countermeasures, the efficiency and accuracy of fault handling are improved.

[0122] In another exemplary embodiment, Figure 3 As shown in the figure, a fault handling method based on log analysis is provided. Figure 1 The system architecture in is taken as an example to illustrate, including the following steps 301 to 309.

[0123] Step 301, classify and preprocess the logs and monitoring indicators to obtain structured data of different categories.

[0124] Step 302 , extracting and clustering features of structured data of different categories to obtain prompt words describing the log content.

[0125] Step 303: According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results.

[0126] Step 304: construct an operation and maintenance knowledge base based on the log tags and analysis results.

[0127] Step 305: construct a fault countermeasure knowledge base, and construct a processing solution agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base.

[0128] Step 306, the processing solution agent automatically matches the fault countermeasure according to the log content and triggers the execution of the corresponding fault countermeasure.

[0129] For specific implementation methods and technical effects of steps 301 to 306 in this embodiment, please refer to Figure 2 The specific description of steps 201 to 206 in the illustrated method embodiment will not be repeated here.

[0130] Step 307, obtaining the operating status, processing effect, and overall performance parameters of the operation and maintenance system.

[0131] Step 308, evaluating the accuracy of the fault countermeasures according to the operation status, processing effect, and overall performance parameters of the operation and maintenance system, and obtaining an evaluation result.

[0132] Step 309, according to the evaluation result, adjust the log content and fault countermeasures associated with the processing solution agent.

[0133] In this embodiment, in order to ensure the quality of the operation and maintenance knowledge base and the fault countermeasures knowledge base, a monitoring and feedback mechanism can be set up, so as to ensure the continuous optimization of the quality and quantity of the knowledge base.

[0134] For example, a comprehensive monitoring and feedback mechanism can be established, such as manual review, indicator setting, and intelligent agent performance verification.

[0135] In this embodiment, the operation status, processing effect and overall performance of the intelligent operation and maintenance system can be monitored, relevant data can be collected and analyzed, and the parameters, strategies or processing solutions of the intelligent agent can be adjusted in time according to the feedback information to improve the efficiency and accuracy of the system.

[0136] For example, manual auditing can be performed during the training process of log automatic classification and intelligent agent construction to prevent leakage of user data during training. In addition, data security auditing can also be set in the automatic screening mechanism to provide security for data processing.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a log analysis-based fault handling device for implementing the log analysis-based fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more log analysis-based fault handling device embodiments provided below can refer to the limitations of the log analysis-based fault handling method above, and will not be repeated here.

[0139] In an exemplary embodiment, Figure 4 As shown, a fault processing device based on log analysis is provided, comprising: a classification and preprocessing module 401, an extraction and clustering module 402, an analysis module 403, an operation and maintenance knowledge base construction module 404, a fault countermeasure knowledge base construction module 405, a processing solution agent construction module 406 and an execution module 407, wherein:

[0140] The classification and preprocessing module 401 is used to classify and preprocess logs and monitoring indicators to obtain structured data of different categories;

[0141] The extraction and clustering module 402 is used to extract features and cluster different types of structured data to obtain prompt words describing the log content;

[0142] The analysis module 403 is used to mark and analyze the log content through the large model according to the prompt word to obtain the log label and analysis result;

[0143] An operation and maintenance knowledge base construction module 404 is used to construct an operation and maintenance knowledge base based on the log tags and the analysis results;

[0144] A fault countermeasure knowledge base construction module 405 is used to construct a fault countermeasure knowledge base;

[0145] A processing solution agent building module 406 is used to build a processing solution agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base;

[0146] The execution module 407 is used to automatically match the fault countermeasures according to the log content through the processing solution agent, and trigger the execution of the corresponding fault countermeasures.

[0147] Exemplarily, the classification and preprocessing module 401 is specifically used to: classify the logs and monitoring indicators into application logs, operation logs, component logs, and host-related monitoring indicators according to the business system and the components on which the business system depends; convert the semi-structured data in the application logs, operation logs, and component logs into structured data to obtain structured data of different categories.

[0148] Exemplarily, the classification and preprocessing module 401 is also used to: extract information from semi-structured data in the application log, operation log, and component log, respectively, to obtain key information, wherein the key information includes: timestamp, event type, source address, and target address in the log; if the application log, operation log, and component log contain user privacy data, encrypt or hide the user privacy data, and retain identification information indicating the user system; store the key information in a preset field format to obtain structured data.

[0149] Exemplarily, the extraction and clustering module 402 is specifically used to: perform data cleaning on structured data of different categories to obtain cleaned data; create a log parsing and clustering pipeline for each category of cleaned data according to the category; split the log through the log parsing and clustering pipeline to obtain several strings of the same length, store the strings in a distributed file system for parallel parsing, and obtain the frequency of occurrence of each string; after sorting the strings from large to small according to the frequency of occurrence, cluster the log content in the strings to obtain K clusters; wherein each cluster corresponds to a log tag, and K is a natural number greater than 0; input the log marked with the log tag into a pre-established prompt word template to generate a prompt word describing the log content.

[0150] Exemplarily, the fault countermeasure knowledge base construction module 405 is specifically used to: construct a fault countermeasure knowledge base based on historical processing experience, expert knowledge, solutions for specific businesses, and preset general solutions; wherein the solutions for specific businesses are jointly established by developers, experts, and operation and maintenance personnel; the general solutions include: alarm templates for monitoring indicators and message notifications of business anomalies.

[0151] Each module in this embodiment can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0152] In another exemplary embodiment, Figure 5 As shown, a fault handling device based on log analysis is provided. Figure 4 Based on the device shown, it can also include:

[0153] The acquisition module 408 is used to obtain the operation status, processing effect, and overall performance parameters of the operation and maintenance system;

[0154] Evaluation module 409, used to evaluate the accuracy of the fault countermeasures according to the operation status, processing effect, and overall performance parameters of the operation and maintenance system, and obtain an evaluation result;

[0155] The adjustment module 410 is used to adjust the log content and fault countermeasures associated with the processing solution agent according to the evaluation result.

[0156] Each module in the above-mentioned fault handling device based on log analysis can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0157] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a fault handling method based on log analysis is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0158] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0161] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0163] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault handling method based on log analysis, characterized in that: The method comprises: Classify and preprocess logs and monitoring indicators to obtain structured data of different categories; Perform feature extraction and clustering on different types of structured data to obtain prompt words describing the log content; According to the prompt words, the log content is marked and analyzed by the large model to obtain log tags and analysis results; Building an operation and maintenance knowledge base based on the log tags and the analysis results; Constructing a fault countermeasure knowledge base, and constructing a processing solution intelligent entity based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; The processing scheme agent automatically matches fault countermeasures according to the log content and triggers the execution of corresponding fault countermeasures.

2. The method according to claim 1, characterized in that The logs and monitoring indicators are classified and preprocessed to obtain structured data of different categories, including: According to the business system and the components that the business system depends on, the logs and monitoring indicators are divided into: application logs, operation logs, component logs, and host-related monitoring indicators; The semi-structured data in the application log, operation log, and component log are converted into structured data respectively to obtain structured data of different categories.

3. The method according to claim 2, characterized in that The converting of the semi-structured data in the application log, the operation log, and the component log into structured data respectively includes: Extracting information from semi-structured data in application logs, operation logs, and component logs to obtain key information, including timestamps, event types, source addresses, and target addresses in the logs. If the application log, operation log, or component log contains user privacy data, encrypt or hide the user privacy data, and retain identification information indicating the user system; The key information is stored in a preset field format to obtain structured data.

4. The method according to claim 1, characterized in that: The feature extraction and clustering of different types of structured data are performed to obtain prompt words describing the log content, including: Perform data cleaning on different categories of structured data to obtain cleaned data; Create log parsing and clustering pipelines for each type of cleaned data by category; The log is split through log parsing and clustering pipeline to obtain several strings of the same length, and the strings are stored in a distributed file system for parallel parsing to obtain the frequency of occurrence of each string; After sorting the strings according to their frequencies of occurrence from large to small, clustering the log contents in the strings to obtain K clusters; each cluster corresponds to a log label, and K is a natural number greater than 0; The logs marked with log tags are input into a pre-established prompt word template to generate prompt words describing the log content.

5. The method according to any one of claims 1 to 4, characterized in that: The construction of the fault countermeasure knowledge base includes: Building a fault countermeasure knowledge base based on historical processing experience, expert knowledge, solutions to unique services, and preset general solutions; wherein the solutions to unique services are jointly established by developers, experts, and operation and maintenance personnel; The general solution includes: alarm templates for monitoring indicators and message notifications of business anomalies.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain the operating status, processing effect, and overall performance parameters of the operation and maintenance system; According to the operation status, processing effect and overall performance parameters of the operation and maintenance system, the accuracy of the fault countermeasures is evaluated to obtain an evaluation result; According to the evaluation result, the log content and fault countermeasures associated with the processing solution agent are adjusted.

7. A fault handling device based on log analysis, characterized in that: The device comprises: The classification and preprocessing module is used to classify and preprocess logs and monitoring indicators to obtain structured data of different categories; The extraction and clustering module is used to extract features and cluster different types of structured data to obtain prompt words describing the log content; An analysis module is used to mark and analyze the log content through a large model according to the prompt words to obtain log tags and analysis results; An operation and maintenance knowledge base construction module, used to construct an operation and maintenance knowledge base based on the log tags and the analysis results; A fault countermeasure knowledge base construction module is used to construct a fault countermeasure knowledge base; A processing solution agent building module, used to build a processing solution agent based on the operation and maintenance knowledge base and the fault countermeasure knowledge base; The execution module is used to automatically match the fault countermeasures according to the log content through the processing solution agent, and trigger the execution of the corresponding fault countermeasures.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images