Log alarm data processing method and device, storage medium and electronic equipment

Abstract

The invention relates to a log alarm data processing method and device, a storage medium and electronic equipment, and relates to the technical field of big data processing.The method comprises the steps that the data source category of original log alarm data is determined, and a key information extraction model corresponding to the data source category is distributed to the original log alarm data; performing semantic analysis on the original log alarm data based on the key information extraction model to obtain alarm key information in the original log alarm data corresponding to the data source category; performing multi-dimensional correlation analysis on the alarm key information of the different data source categories based on a preset correlation analysis model to obtain a correlation relationship between the alarm key information of the different data source categories; and performing alarm root cause reasoning on the association relationship between the alarm key information based on a preset fault reasoning model to obtain an alarm association analysis result. According to the invention, the accuracy of the alarm association analysis result is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of big data processing. Specifically, the present disclosure relates to a method for processing log alarm data, an apparatus for processing log alarm data, a computer-readable storage medium, and an electronic device. Background Art

[0002] In traditional methods for processing log alarm data, the alarm data from different data sources are independent of each other, thus easily forming information silos. On this premise, if it is necessary to perform correlation analysis on the alarm data from different data sources, it is necessary for operation and maintenance personnel to achieve this manually. However, this method has problems of low analysis efficiency and low accuracy of analysis results.

[0003] It should be noted that the information disclosed in the above background art is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a method for processing log alarm data, an apparatus for processing log alarm data, a computer-readable storage medium, and an electronic device, thereby at least to a certain extent overcoming the problems of low analysis efficiency and low accuracy of analysis results caused by the limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, there is provided a method for processing log alarm data, including:

[0006] Determine the data source category of the original log alarm data, and allocate a key information extraction model corresponding to the data source category to the original log alarm data;

[0007] Perform semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category;

[0008] Perform multi-dimensional correlation analysis on the alarm key information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the alarm key information of different data source categories;

[0009] Perform alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault reasoning model to obtain an alarm correlation analysis result.

[0010] In an exemplary embodiment of the present disclosure, determining the data source category of the original log alarm data includes:

[0011] Determine the data source category of the original log alert data according to the data source channels of the original log alert data; wherein, the data source category includes at least one of the application log category, the system log category, the monitoring metric category, and other data source categories; the other data source categories include at least one of the third-party service log category, the security log category, and the operation log category;

[0012] The key information extraction model includes at least one of a first key information extraction model corresponding to the application log category, a second key information extraction model corresponding to the system log category, a third key information extraction model corresponding to the monitoring metric category, and a fourth key information extraction model corresponding to other data source categories.

[0013] In an exemplary embodiment of the present disclosure, semantic analysis is performed on the original log alert data based on the key information extraction model to obtain the alert key information in the original log alert data corresponding to the data source category, including:

[0014] Based on the first key information extraction model, call the application service interface to obtain the original application log alert data corresponding to the application log category, and perform semantic analysis on the original application log alert data to obtain the first alert key information in the original application log alert data;

[0015] Based on the second key information extraction model, call the system log collection tool to obtain the original system log alert data corresponding to the system log category, and perform semantic analysis on the original system log alert data to obtain the second alert key information in the original system log alert data;

[0016] Based on the third key information extraction model, call the status monitoring tool to obtain the original monitoring metric alert data corresponding to the monitoring metric category, and perform semantic analysis on the original monitoring metric alert data to obtain the third alert key information in the original monitoring metric alert data;

[0017] Based on the fourth key information extraction model, call other log collection tools to obtain the original other log data corresponding to other data source categories, and perform speech analysis on the original other log data to obtain the fourth alert key information in the original other log data.

[0018] In an exemplary embodiment of the present disclosure, the first key information extraction model includes a first embedding mapping layer, a first encoding layer, and a first mixture-of-experts model layer;

[0019] Among them, performing semantic analysis on the original application log alert data based on the first key information extraction model to obtain the first alert key information in the original application log alert data includes:

[0020] Generate the first basic information to be predicted based on the program running status information and program running error information in the original application log alarm data, and generate the first context information to be predicted based on the program running event information in the original application log alarm data and the preset first model prompt parameters;

[0021] Perform embedding mapping processing on the first basic information to be predicted based on the first embedding mapping layer to obtain the first log feature, and perform embedding mapping processing on the first context information to be predicted based on the first embedding mapping layer to obtain the first context flag sequence;

[0022] Perform encoding processing on the first log feature and the first context flag sequence based on the first encoding layer to obtain the first overall context representation, and perform semantic analysis on the first context flag sequence and the first overall context representation based on the first mixture-of-experts model layer to obtain the first alarm key information in the original application log alarm data.

[0023] In an exemplary embodiment of the present disclosure, the first mixture-of-experts model layer includes a first gated network model and a plurality of first expert neural network models;

[0024] Among them, performing semantic analysis on the first context flag sequence and the first overall context representation based on the first mixture-of-experts model layer to obtain the first alarm key information in the original application log alarm data includes:

[0025] Based on the first gated network model, determine the first model weight of the first expert neural network model in the alarm type dimension, the second model weight in the alarm level dimension, the third model weight in the alarm time dimension, and the fourth model weight in the alarm-related component dimension according to the first context flag sequence;

[0026] Based on the first model weight, the second model weight, the third model weight, and the fourth model weight, determine the first sub-goal neural network model required to perform the semantic analysis task in the alarm type dimension, the second sub-goal neural network model required to perform the semantic analysis task in the alarm level dimension, the third sub-goal neural network model required to perform the semantic analysis task in the alarm time dimension, and the fourth sub-goal neural network model required to perform the semantic analysis task in the alarm-related component dimension from the plurality of first expert neural network models;

[0027] Input the first context flag sequence and the overall first context representation into the first target neural network model, the second target neural network model, the third target neural network model, and the fourth target neural network model respectively, to obtain the first semantic prediction result in the alarm type dimension, the second semantic prediction result in the alarm level dimension, the third semantic prediction result in the alarm time dimension, and the fourth semantic prediction result in the dimension of components involved in the alarm;

[0028] Generate the first alarm key information in the original application log alarm data according to the first semantic prediction result, the second semantic prediction result, the third semantic prediction result, and the fourth semantic prediction result.

[0029] In an exemplary embodiment of the present disclosure, the method for processing the log alarm data further includes:

[0030] Determine the alarm category corresponding to the original application log alarm data according to the first alarm key information, and determine the alarm priority corresponding to the original application log alarm data according to the alarm category;

[0031] If the alarm priority reaches the preset priority condition, determine the component routing path according to the fourth semantic prediction result in the dimension of components involved in the alarm included in the first alarm key information;

[0032] Based on the component routing path, determine the operation instruction corresponding to the alarm category, and execute the operation instruction to solve the abnormal behavior that generates the original application log alarm data.

[0033] In an exemplary embodiment of the present disclosure, the preset association analysis model includes a second embedding mapping layer, a second encoding layer, and a second mixture-of-experts model layer;

[0034] Among them, performing multi-dimensional association analysis on the alarm key information of different data source categories based on the preset association analysis model to obtain the association relationship between the alarm key information of different data source categories includes:

[0035] Generate the basic key information to be predicted according to the alarm key information of different data source categories, and generate the context key information to be predicted according to the preset second model hint parameters;

[0036] Perform embedding mapping processing on the basic key information to be predicted based on the second embedding mapping layer to obtain key information features, and perform embedding mapping processing on the context key information to be predicted based on the second embedding mapping layer to obtain a key context flag sequence;

[0037] Encoding the key information features and the key context flag sequence based on the second encoding layer to obtain an overall key context representation, and performing multi-dimensional correlation analysis on the key context flag sequence and the overall key context representation based on the second mixture-of-experts model layer to obtain the correlation relationship between the alarm key information of different data source categories.

[0038] In an exemplary embodiment of the present disclosure, the second mixture-of-experts model layer includes a second gating network model and a plurality of second expert neural network models;

[0039] Among them, performing multi-dimensional correlation analysis on the key context flag sequence and the overall key context representation based on the second mixture-of-experts model layer to obtain the correlation relationship between the alarm key information of different data source categories includes:

[0040] Based on the second gating network model and according to the key context flag sequence, determining a fifth model weight in the time relationship dimension, a sixth model weight in the space relationship dimension, and a seventh model weight in the causal relationship dimension of the second expert neural network model;

[0041] Based on the fifth model weight, the sixth model weight, and the seventh model weight, determining a fifth sub-goal neural network model required for performing the correlation analysis task in the time relationship dimension, a sixth sub-goal neural network model required for performing the correlation analysis task in the space relationship dimension, and a seventh sub-goal neural network model required for performing the correlation analysis task in the causal relationship dimension from a plurality of second expert neural network models;

[0042] Inputting the key context flag sequence and the overall key context representation into a fifth target neural network model, a sixth target neural network model, and a seventh target neural network model respectively to obtain a first relationship prediction result in the time relationship, a second relationship prediction result in the space relationship dimension, and a third relationship prediction result in the causal relationship dimension;

[0043] Generating the correlation relationship between the alarm key information of different data source categories according to the first relationship prediction result, the second relationship prediction result, and the third relationship prediction result.

[0044] In an exemplary embodiment of the present disclosure, performing alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault inference model to obtain an alarm correlation analysis result, including:

[0045] Generating association basic information to be predicted according to the correlation relationship between the alarm key information, and generating association context information to be predicted according to a preset third model prompt parameter;

[0046] Input the to-be-predicted associated basic information and the to-be-predicted associated context information into a preset fault reasoning model for alarm root cause reasoning to obtain an alarm association analysis result;

[0047] Among them, the alarm association analysis result includes the time relationship, spatial relationship, causal relationship between alarms, the description information of the fault root cause, and the fault solution for solving the fault.

[0048] According to one aspect of the present disclosure, there is provided a processing device for log alarm data, including:

[0049] An information extraction model determination module, configured to determine the data source category of the original log alarm data, and allocate a key information extraction model corresponding to the data source category to the original log alarm data;

[0050] An alarm key information determination module, configured to perform semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category;

[0051] An association relationship analysis module, configured to perform multi-dimensional association analysis on the alarm key information of different data source categories based on a preset association analysis model to obtain the association relationship between the alarm key information of different data source categories;

[0052] An alarm root cause reasoning module, configured to perform alarm root cause reasoning on the association relationship between the alarm key information based on a preset fault reasoning model to obtain an alarm association analysis result.

[0053] According to one aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the processing method for log alarm data described in any one of the above is implemented.

[0054] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0055] A processor; and

[0056] A memory, configured to store the executable instructions of the processor;

[0057] Among them, the processor is configured to execute the processing method for log alarm data described in any one of the above by executing the executable instructions.

[0058] A method for processing log alarm data provided by an embodiment of the present disclosure, on the one hand, determines the data source category of the original log alarm data, and assigns a key information extraction model corresponding to the data source category to the original log alarm data; then performs semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category; further performs multi-dimensional correlation analysis on the alarm key information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the alarm key information of different data source categories; finally, performs alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault reasoning model to obtain the alarm correlation analysis result, realizing the correlation analysis of alarm data of multiple different data source categories and improving the analysis efficiency of alarm data; on the other hand, also improves the accuracy of the obtained alarm correlation analysis result.

[0059] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0061] Figure 1 Schematically shows a flowchart of a method for processing log alarm data according to an exemplary embodiment of the present disclosure.

[0062] Figure 2 Schematically shows a structural example diagram of a log alarm data processing system according to an exemplary embodiment of the present disclosure.

[0063] Figure 3 Schematically shows a structural example diagram of a key information extraction model according to an exemplary embodiment of the present disclosure.

[0064] Figure 4 Schematically shows a structural diagram of a correlation analysis model according to an exemplary embodiment of the present disclosure.

[0065] Figure 5 Schematically shows an example diagram of a matching scenario of an information extraction model according to an exemplary embodiment of the present disclosure.

[0066] Figure 6 Schematically shows an example diagram of an obtained correlation chart and / or correlation report according to an exemplary embodiment of the present disclosure.

[0067] Figure 7 Schematically shows a structural example diagram of a processing device for log alarm data according to an exemplary embodiment of the present disclosure.

[0068] Figure 8 Schematically shows an electronic device for implementing a method for processing log alarm data according to an exemplary embodiment of the present disclosure. Detailed implementation manners

[0069] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be used. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring the various aspects of the present disclosure.

[0070] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0071] The specific processing process of traditional log alarm data can usually be implemented based on the following several methods: One is to trigger an alarm based on predefined rules or thresholds; that is, in the actual application process, when the system has an anomaly, the monitoring system will send an alarm notification according to the preset rules; however, these systems usually lack the ability of intelligent analysis, cannot handle complex alarm correlation relationships, and are prone to generating a large number of noisy alarms, making it difficult for operation and maintenance personnel to quickly locate problems; Another is to implement it based on log aggregation and analysis tools; specifically, for systems such as ELK (Elasticsearch, Logstash, and Kibana), the log aggregation and analysis tools can be used to centrally store, retrieve, and visually analyze the logs of products during the operation of the system; however, these tools usually require users to manually define query rules and analysis logic, cannot automatically discover the correlation relationships between alarms, and still rely on the experience of operation and maintenance personnel; Another one is to perform log alarm data analysis based on natural language processing (NLP, Natural Language Processing); specifically, in the actual application process, NLP technology can be used for log parsing, extracting key information, or performing simple alarm classification, etc. However, these solutions usually only use a part of the capabilities of NLP technology, do not fully utilize the powerful reasoning and semantic understanding capabilities of LLM, and do not adopt the mode of multi-Agent collaborative work, and cannot perform in-depth correlation analysis on multi-dimensional alarm information.

[0072] Based on the above-recorded content, it can be known that the processing methods of log alarm data recorded in the existing solutions mainly have the following defects: On the one hand, the correlation analysis ability is weak; that is, the existing log analysis tools rely on manual rule configuration, lack the ability of intelligent correlation analysis and cannot handle complex alarm relationships; On the other hand, it is difficult to locate the root cause; that is, the existing methods are difficult to quickly locate the root cause of the fault, resulting in too long fault handling time and affecting business continuity; On the other hand, the alarm handling ability is weak; that is, the existing methods cannot take immediate actions when an alarm occurs, often need to wait for the analysis results of the large model, and cannot notify users in real time; On the other hand, the adaptability ability is weak; that is, the existing methods rely on predefined rules or knowledge bases and cannot adapt to new fault modes; Finally, the operation and maintenance efficiency is low; that is, the existing methods need to spend a lot of time and effort in handling and analyzing alarms, with low efficiency.

[0073] Based on this, in the exemplary embodiment of the present example, a method for processing log alarm data is first provided, and this method can run on a terminal device, a server, a server cluster, a cloud server, etc.; of course, those skilled in the art can also run the method of the present disclosure on other platforms according to needs, and no special limitation is made in this exemplary embodiment. Specifically, refer to Figure 1As shown, the method for processing log alarm data may include the following steps:

[0074] Step S110. Determine the data source category of the original log alarm data, and assign a key information extraction model corresponding to the data source category to the original log alarm data;

[0075] Step S120. Perform semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category;

[0076] Step S130. Perform multi-dimensional correlation analysis on the alarm key information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the alarm key information of different data source categories;

[0077] Step S140. Perform alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault reasoning model to obtain the alarm correlation analysis result.

[0078] In the method for processing log alarm data described above, on the one hand, by determining the data source category of the original log alarm data and assigning a key information extraction model corresponding to the data source category to the original log alarm data; then performing semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category; further performing multi-dimensional correlation analysis on the alarm key information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the alarm key information of different data source categories; finally performing alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault reasoning model to obtain the alarm correlation analysis result, the correlation analysis of alarm data of multiple different data source categories is realized, and the analysis efficiency of the alarm data is improved; on the other hand, the accuracy of the obtained alarm correlation analysis result is also improved.

[0079] Hereinafter, the method for processing log alarm data recorded in the exemplary embodiments of the present disclosure will be explained and described in detail with reference to the accompanying drawings.

[0080] First, the nouns involved in the exemplary embodiments of the present disclosure will be explained and described.

[0081] LLM (Large Language Model): A large language model refers to a deep learning model trained with a large amount of text data, which can understand and generate natural language text and has capabilities such as semantic understanding, text generation, and reasoning.

[0082] Multi-agent system: A system composed of multiple autonomous agents. Each agent in the system has specific tasks and goals, and they can cooperate with each other to complete complex tasks. In the multi-agent system involved in the exemplary embodiments of the present disclosure, each agent included in the system can be responsible for the log analysis of different data categories and interact with other systems.

[0083] Multi-dimensional alarm: Refers to alarm information from different data source categories (e.g., application logs, system logs, monitoring metrics).

[0084] Correlation analysis: Refers to the analysis of multiple alarm messages to discover the correlation relationships between them, such as causal relationships, temporal relationships, spatial relationships, etc., so as to locate the root cause of the alarm problem.

[0085] Secondly, the technical implementation principle of the exemplary embodiments of the present disclosure will be explained and described. Specifically, the method for processing log alarm data recorded in the exemplary embodiments of the present disclosure can efficiently and accurately locate the root cause of alarms in complex systems, reducing the time and cost of manual troubleshooting. At the same time, in the specific application process, a powerful semantic understanding and reasoning ability of the large language model and a collaborative working mode of multi-agent can be combined to construct an intelligent system for processing log alarm data. Based on this system, problems such as high alarm noise, weak correlation analysis ability, difficult root cause location, and insufficient alarm real-time performance that occur in traditional alarm systems in a complex information environment can be solved. On this premise, the method for processing log alarm data recorded in the exemplary embodiments of the present disclosure can be carried out around the following aspects: On the one hand, semantic understanding based on LLM; that is, in the specific process of processing log alarm data, the LLM can be used to perform semantic analysis on alarm data to understand key information such as alarm type, level, occurrence time, and involved components, so that the system can "understand" the meaning of log alarms. On the other hand, reasoning and correlation based on LLM; that is, in the specific process of processing log alarm data, the reasoning ability of the LLM can be used, combined with multi-dimensional analysis methods, to automatically identify causal relationships, temporal relationships, etc. between alarms and infer the root cause of the failure. On the further hand, autonomous collaborative work based on Multi-agent; that is, in the specific process of processing log alarm data, the system can be divided into multiple alarm analysis Agents and one correlation analysis Agent. Among them, each alarm analysis Agent is responsible for processing alarm data from specific data sources, and has the ability to directly read data sources, execute Action actions (such as alarm classification and standardization), store key information, and issue independent alarms, and can execute corresponding processing modules according to the alarm type. At the same time, the correlation analysis Agent can be used to be responsible for global alarm correlation analysis and root cause location. In the actual application process, each Agent can work independently or cooperate with each other to jointly complete alarm analysis and root cause location. On still another hand, direct data processing based on LLM; that is, in the specific process of processing log alarm data, each alarm analysis Agent can be used to directly read the original alarm data from different data sources, and then use the LLM for parsing, semantic analysis, and processing, simplifying the data processing flow.

[0086] Further, the system for processing log alarm data involved in the exemplary embodiments of the present disclosure will be explained and described. Specifically, refer to Figure 2As shown in the figure, the processing system for log alert data may include a multi-source log alert analysis agent 210, a communication component 220, a joint analysis and decision-making agent 230, a database 240, and a data visualization component 250. Among them, the multi-source log alert analysis agent is communicatively connected to the joint analysis and decision-making agent through the communication component. The joint analysis and decision-making agent is communicatively connected to the database and the data visualization component respectively. The multi-source log alert analysis agent is communicatively connected to the database. In the actual application process, the multi-source log alert analysis agent can be used to analyze the original log alert data with different data source categories to obtain key alert information. The communication component is responsible for sending the key alert information to the joint analysis and decision-making agent. The joint analysis and decision-making agent is used to analyze the correlation relationship between the key alert information to obtain an alert correlation analysis result. The data visualization component is used to display the alert correlation analysis result, and the database is used to store the key alert information and the alert correlation analysis result. At the same time, in the actual application process, it is also necessary to perform initialization processing on the database to ensure the reliability and security of data storage. Further, it is also necessary to configure the data visualization component to support custom visualization configuration, which is convenient for users to adjust according to their needs.

[0087] In an example embodiment, in the actual application process, in order to analyze the log alert data of different data source categories, first, it is necessary to select a suitable LLM model and perform deployment and integration. Second, for different data sources, develop corresponding alert analysis Agents. Among them, the alert analysis Agent needs to have the following functions: on the one hand, integrate the LLM for semantic analysis to extract key alert information so as to achieve the purpose of implementing Action actions such as alert classification, priority judgment, standardization, routing, and suppression. On the other hand, the alert analysis Agent also needs to have local data storage and independent alert capabilities, and also needs to be able to directly read the data source without preprocessing the data source. On the other hand, the alert analysis Agent also needs to execute corresponding processing modules according to the alert type. Then, develop an association analysis Agent (i.e., the joint analysis and decision-making agent), use the LLM for multi-dimensional association analysis, determine the relationship between alerts, and locate the root cause of the failure. Further, it is also necessary to develop an Agent cooperation mechanism. That is, based on the Agent cooperation mechanism, communication and cooperation mechanisms between Agents can be realized to ensure the efficient operation of Agents. Furthermore, it is also necessary to develop a module that can store log alert data and analysis results and realize the effective management of data.

[0088] In an exemplary embodiment, in the actual process of developing an agent, first of all, selecting a suitable LLM model, designing an effective Prompt, and performing domain fine-tuning are the keys to ensuring analysis accuracy; secondly, the ability and efficiency of the alarm analysis Agent to directly read data sources also need to be considered; that is, in the actual application process, how to ensure that the LLMAgent can efficiently read various data sources is the key to improving system efficiency; furthermore, designing an efficient Agent collaboration mechanism is the key to ensuring the overall performance of the system; and designing an effective multi-dimensional correlation analysis algorithm is the key to achieving accurate fault location; furthermore, implementing fine-grained processing, real-time notification, and flexible processing of alarms is also the key to improving the real-time performance, flexibility, and adaptive ability of the system.

[0089] It should also be noted here that in the actual development process, the following aspects of problems need to be paid attention to: on the one hand, how to give full play to the performance of the LLM and solve the problem of computing resource consumption of the LLM in actual applications; on the other hand, how to ensure that the LLMAgent can connect to different data sources and can efficiently and stably read data; on the other hand, how to effectively manage and coordinate multiple Agents to ensure that each agent can work together and ensure the stability and performance of the system; on the other hand, how to perform correlation analysis in multiple dimensions and ensure the accuracy and efficiency of the analysis; finally, how to ensure the effective execution of Action actions, the reliability of independent alarms, and the ability to execute corresponding processing modules according to the alarm type.

[0090] In an exemplary embodiment, the functions of the above-mentioned alarm analysis agent are as follows: each Agent is responsible for processing alarm data from a specific data source, directly reads the original data of the corresponding data source, uses the LLM to perform semantic analysis on the alarm data, extracts key alarm information (including alarm type, alarm level, occurrence time, related components, etc.), and performs Action actions such as alarm classification, priority judgment, standardization, routing, and suppression; and according to the alarm type, execute the corresponding processing module; at the same time, store the extracted key alarm information in the local data storage module, and when the alarm level reaches a specific threshold, independently send an alarm notification to the user; the specific functions to be executed are: data parsing, LLM semantic analysis, extraction of key alarm information, alarm classification, alarm priority judgment, alarm standardization, alarm routing, alarm suppression, execution of corresponding processing modules, local storage of key alarm information, and independent alarm to users, etc. Among them, the related components recorded here refer to the source components of the alarm, such as Kafka, ES, K8s, or Hive, etc.

[0091] In an exemplary embodiment, the specific role of the joint analysis decision-making agent described above is as follows: receiving the key alarm information from the alarm analysis Agent, performing multi-dimensional correlation analysis using the LLM to identify causal relationships, temporal relationships, etc. between alarms, and finally using the LLM to reason and locate the root cause of the fault. The specific functions to be executed are: LLM reasoning analysis, alarm correlation relationship identification, root cause of fault location, and providing suggestions for solutions.

[0092] In an exemplary embodiment, the specific role of the database described above is: storing the original log alarm data, key alarm information, and correlation analysis results; the specific role of the communication component described above is: realizing data transfer between each Agent; the role of the data visualization component described above is: visually displaying the analysis results in the form of charts, reports, etc.

[0093] Hereinafter, the model structure of the first key information extraction model described in the exemplary embodiments of the present disclosure will be explained and described. Specifically, referring to Figure 3 As shown, the first key information extraction model may include a first input layer 301, a first embedding mapping layer 302, a first encoding layer 303, a first mixture-of-experts model layer 304, and a first output layer 305; wherein, the first mixture-of-experts model layer described here includes a first gating network model and multiple first expert neural network models. At the same time, the roles played by each model layer in the process of key information extraction will be described in detail later, and no further elaboration will be made here.

[0094] Hereinafter, the model structure of the correlation analysis model described in the exemplary embodiments of the present disclosure will be explained and described. Specifically, referring to Figure 4 As shown, the preset correlation analysis model described here may include a second input layer 401, a second embedding mapping layer 402, a second encoding layer 403, a second mixture-of-experts model layer 404, and a second output layer 405; wherein, the second mixture-of-experts model layer described here includes a second gating network model and multiple second expert neural network models. At the same time, the roles played by each model layer in the process of correlation analysis will be described in detail later, and no further elaboration will be made here. It should be added here that the model structure of the key information extraction model is generally similar to that of the correlation analysis model, and the possible difference is the number of gating network models specifically used, because the dimensions considered in the process of key information extraction are different from those considered in the process of correlation analysis.

[0095] It should also be further supplemented and explained here that in the process of fine-tuning the key information extraction model and the correlation analysis model, different fine-tuning methods such as full-parameter fine-tuning, LoRA (Low-Rank Adaptation), and Prompt Tuning can be used to achieve it; at the same time, the training data used in the model fine-tuning process can be achieved by using alarm data or specific fault scenario data in the field. And by fine-tuning the model, the accuracy of the obtained correlation relationship can be further improved.

[0096] Next, in combination with Figures 2 - 4 to Figure 1 the processing method of the log alarm data shown in will be further explained and described. Specifically:

[0097] In step S110, determine the data source category of the original log alarm data, and allocate a key information extraction model corresponding to the data source category to the original log alarm data.

[0098] In this exemplary embodiment, first, determine the data source category of the original log alarm data; specifically, it can be achieved in the following manner: according to the data source channel of the original log alarm data, determine the data source category of the original log alarm data; wherein, the data source categories recorded here may include but are not limited to application log categories, system log categories, monitoring metric categories, and other data source categories, etc., and the other data source categories recorded here may include but are not limited to third-party service log categories, security log categories, and operation log categories, etc.; secondly, allocate a key information extraction model corresponding to the data source category to the original log alarm data; specifically, the key information extraction models recorded here may include a first key information extraction model corresponding to the application log category, a second key information extraction model corresponding to the system log category, a third key information extraction model corresponding to the monitoring metric category, and a fourth key information extraction model corresponding to the other data source categories, etc.; among them, for the matching scenario diagram of the specific information extraction model, reference can be made to Figure 5 shown.

[0099] In step S120, based on the key information extraction model, perform semantic analysis on the original log alarm data to obtain the alarm key information in the original log alarm data corresponding to the data source category.

[0100] Specifically, the specific determination process of the alarm key information can be implemented in the following manner: Invoke the application service interface based on the first key information extraction model to obtain the original application log alarm data corresponding to the application log category, and perform semantic analysis on the original application log alarm data to obtain the first alarm key information in the original application log alarm data; Invoke the system log collection tool based on the second key information extraction model to obtain the original system log alarm data corresponding to the system log category, and perform semantic analysis on the original system log alarm data to obtain the second alarm key information in the original system log alarm data; Invoke the status monitoring tool based on the third key information extraction model to obtain the original monitoring metric alarm data corresponding to the monitoring metric category, and perform semantic analysis on the original monitoring metric alarm data to obtain the third alarm key information in the original monitoring metric alarm data; Invoke other log collection tools based on the fourth key information extraction model to obtain the original other log data corresponding to the other data source category, and perform speech analysis on the original other log data to obtain the fourth alarm key information in the original other log data. That is, in the actual application process, the original log data of different data source categories can be extracted respectively based on different key information extraction models, and then the corresponding original log alarm data can be extracted from the original log data.

[0101] In an exemplary embodiment, the above-mentioned original application log alarm data is generated during the operation of the application program, which can record the abnormal operation status information, error information, and abnormal events, etc. of the application program during operation; in the actual application process, the original application log alarm data has the characteristics of inconsistent data formats, complex information, and diverse semantics. Therefore, it is necessary to extract the first alarm key information in the original application log alarm data to facilitate subsequent correlation analysis; if the correlation analysis is directly based on the original application log alarm data, there will be problems such as heavy system burden and low accuracy of the obtained correlation analysis results.

[0102] In an exemplary embodiment, the above-mentioned original system log alarm data is generated by the operating system kernel or service program, which can record the abnormal operation status information of the system during operation, abnormal event information on the hardware, and system error information, etc.; in the actual application process, although the data format of the original system log alarm data is relatively standardized, the amount of information is huge and requires in-depth analysis. Therefore, it is necessary to extract the second alarm key information in the original system log alarm data to facilitate subsequent correlation analysis; if the correlation analysis is directly based on the original system log alarm data, there will be a problem of excessive calculation volume.

[0103] In an exemplary embodiment, the above-described original monitoring metric alert data can be collected by a monitoring system, which can be used to record multiple different metrics such as abnormal CPU usage, abnormal memory usage, and abnormal network traffic of the system; in the actual application process, although the original monitoring metric alert data has a structured data format, it has noise and fluctuations, so it is necessary to extract the third key alert information from the original monitoring metric alert data for subsequent correlation analysis; if the correlation analysis is directly based on the original monitoring metric alert data, there will be a problem that the noise is too large and the accuracy of the analysis result is low.

[0104] In an exemplary embodiment, the above-described original other log data may include, but is not limited to, third-party service logs, security logs, and operation logs, etc.; among them, the third-party service logs can be collected through third-party log collection tools, the security logs refer to abnormal security log information generated during the operation of the application program, and the operation logs refer to abnormal operation log information generated during the operation of the application program; in the actual application process, although the amount of the original other log data is small, its sources are extensive and the data formats are not unified, so it is necessary to extract the fourth key alert information from the original other log data for subsequent correlation analysis.

[0105] In an exemplary embodiment, semantic analysis is performed on the original application log alert data based on the first key information extraction model to obtain the first alert key information in the original application log alert data, which can be achieved in the following manner: generating first basic information to be predicted according to the program running status information and program running error information in the original application log alert data, and generating first context information to be predicted according to the program running event information in the original application log alert data and the preset first model prompt parameters; performing embedding mapping processing on the first basic information to be predicted based on the first embedding mapping layer to obtain first log features, and performing embedding mapping processing on the first context information to be predicted based on the first embedding mapping layer to obtain a first context flag sequence; performing encoding processing on the first log features and the first context flag sequence based on the first encoding layer to obtain a first overall context representation, and performing semantic analysis on the first context flag sequence and the first overall context representation based on the first mixture-of-experts model layer to obtain the first alert key information in the original application log alert data. Among them, the first embedding mapping layer described here may include an Embedding embedding mapping layer and a Bert embedding mapping layer. In the actual application process, embedding mapping processing can be performed on the first basic information to be predicted based on the Embedding embedding mapping layer to obtain first log features; embedding mapping processing can also be performed on the first context information to be predicted based on the Bert embedding mapping layer to obtain a first context flag sequence; further, the first encoding layer described here can be a bidirectional multi-layer Transformer, and of course other structures can also be selected, and this example does not make special restrictions on this.

[0106] In an exemplary embodiment, semantic analysis is performed on the first context flag sequence and the first overall context representation based on the first mixture-of-experts model layer to obtain the first key alarm information in the original application log alarm data, which can be achieved in the following manner: Based on the first gating network model, according to the first context flag sequence, determine the first model weight of the first expert neural network model in the alarm type dimension, the second model weight in the alarm level dimension, the third model weight in the alarm time dimension, and the fourth model weight in the dimension of components involved in the alarm; Based on the first model weight, the second model weight, the third model weight, and the fourth model weight, determine the first sub-goal neural network model required to perform the semantic analysis task in the alarm type dimension, the second sub-goal neural network model required to perform the semantic analysis task in the alarm level dimension, the third sub-goal neural network model required to perform the semantic analysis task in the alarm time dimension, and the fourth sub-goal neural network model required to perform the semantic analysis task in the dimension of components involved in the alarm from multiple first expert neural network models; Input the first context flag sequence and the first overall context representation into the first target neural network model, the second target neural network model, the third target neural network model, and the fourth target neural network model respectively to obtain the first semantic prediction result in the alarm type dimension, the second semantic prediction result in the alarm level dimension, the third semantic prediction result in the alarm time dimension, and the fourth semantic prediction result in the dimension of components involved in the alarm; Generate the first key alarm information in the original application log alarm data according to the first semantic prediction result, the second semantic prediction result, the third semantic prediction result, and the fourth semantic prediction result. That is, in the actual process of predicting the first key alarm information, prediction can be performed from multiple dimensions such as the alarm type, the alarm level, the alarm occurrence time, and the components involved in the alarm.

[0107] Further, after obtaining the first key warning information, the method for processing the log warning data further includes: determining the warning category corresponding to the original application log warning data according to the first key warning information, and determining the warning priority corresponding to the original application log warning data according to the warning category; if the warning priority reaches the preset priority condition, determining the component routing path according to the fourth semantic prediction result in the component dimension involved in the warning included in the first key warning information; determining an operation instruction corresponding to the warning category based on the component routing path, and executing the operation instruction to resolve the abnormal behavior that generates the original application log warning data. That is, in the actual application process, operations such as warning classification, priority judgment, standardization, routing, and suppression can be performed according to the analysis result (i.e., the first key warning information), and then the corresponding processing module is executed according to the warning type; finally, the extracted key warning information is stored in the local data storage module, and when the warning level reaches a specific threshold, an independent warning notification is sent to the user. At the same time, the suppression operation described here refers to filtering and suppressing duplicate, similar warning type, or less important warning data to reduce warning noise; among them, the specific suppression method can be implemented based on the warning occurrence time, the number of warnings, and the similarity of the warning content; the processing module described here can be an internal system module (such as an automatic restart module or a data cleaning module, etc.), or an external service module (such as a work order system or a knowledge base system, etc.); at the same time, the internal module described here refers to a module predefined inside the system, for example: an automatic restart module, which can be used to automatically restart a faulty service or component; another example is a data cleaning module, which can be used to clean invalid or incorrect data; another example is a security isolation module, which can be used to isolate components with security risks; another example is a resource adjustment module, which can be used to adjust system resource allocation, such as increasing CPU or memory, etc. Further, the external service module described here can be a third-party service outside the system; for example, a work order system, which can be used to automatically create work orders and notify the operation and maintenance personnel; it can also be a knowledge base system, which can be used to query fault solutions; it can also be an automated operation and maintenance platform, which is used to execute more complex automated operation and maintenance tasks.

[0108] It should also be further supplemented and explained here that the specific model structures of the second, third, and fourth key information extraction models described above are the same as those of the first key information extraction model, and the specific extraction processes of the second, third, and fourth alarm key information are similar to that of the first alarm key information, so no further elaboration will be provided here. At the same time, when the second, third, and fourth key information extraction models detect that the alarm level reaches a specific threshold, they can also independently send alarm notifications to users. At the same time, the specific sending method of the alarm notification can be based on emails, text messages, or instant messaging tools, etc., and this example does not impose special restrictions on this.

[0109] In step S130, based on a preset correlation analysis model, multi-dimensional correlation analysis is performed on the alarm key information of different data source categories to obtain the correlation relationships between the alarm key information of different data source categories.

[0110] Specifically, the specific determination process of the correlation relationships between the alarm key information of different data source categories can be achieved through the following method: Generate the basic key information to be predicted based on the alarm key information of different data source categories, and generate the context key information to be predicted based on the preset second model prompt parameters; Perform embedding mapping processing on the basic key information to be predicted based on the second embedding mapping layer to obtain key information features, and perform embedding mapping processing on the context key information to be predicted based on the second embedding mapping layer to obtain the key context flag sequence; Perform encoding processing on the key information features and the key context flag sequence based on the second encoding layer to obtain the overall key context representation, and perform multi-dimensional correlation analysis on the key context flag sequence and the overall key context representation based on the second mixture-of-experts model layer to obtain the correlation relationships between the alarm key information of different data source categories.

[0111] In an exemplary embodiment, based on the second mixture-of-experts model layer, a multi-dimensional correlation analysis is performed on the key context flag sequence and the overall key context representation to obtain the correlation relationship between the alarm key information of different data source categories, which can be achieved in the following manner: Based on the second gating network model, according to the key context flag sequence, determine the fifth model weight of the second expert neural network model in the time relationship dimension, the sixth model weight in the spatial relationship dimension, and the seventh model weight in the causal relationship dimension; Based on the fifth model weight, the sixth model weight, and the seventh model weight, determine the fifth sub-goal neural network model required to perform the correlation analysis task in the time relationship dimension, the sixth sub-goal neural network model required to perform the correlation analysis task in the spatial relationship dimension, and the seventh sub-goal neural network model required to perform the correlation analysis task in the causal relationship dimension from multiple second expert neural network models; Input the key context flag sequence and the overall key context representation into the fifth target neural network model, the sixth target neural network model, and the seventh target neural network model respectively to obtain the first relationship prediction result in the time relationship, the second relationship prediction result in the spatial relationship dimension, and the third relationship prediction result in the causal relationship dimension; Generate the correlation relationship between the alarm key information of different data source categories according to the first relationship prediction result, the second relationship prediction result, and the third relationship prediction result; That is to say, in the actual application process, the correlation dimension between the alarm key information can be determined from multiple different dimensions such as the time dimension, the spatial dimension, and the causal dimension. Of course, other dimensions can also be considered, such as the semantic dimension, etc. This example does not make special restrictions on this.

[0112] In step S140, based on the preset fault inference model, an alarm root cause inference is performed on the correlation relationship between the alarm key information to obtain an alarm correlation analysis result.

[0113] Specifically, the specific determination process of the alarm correlation analysis result can be implemented in the following way: generating the correlation basic information to be predicted according to the correlation relationship between the alarm key information, and generating the correlation context information to be predicted according to the preset third model prompt parameter; inputting the correlation basic information to be predicted and the correlation context information to be predicted into the preset fault reasoning model for alarm root cause reasoning to obtain the alarm correlation analysis result; wherein, the alarm correlation analysis result includes the time relationship, spatial relationship, causal relationship between alarms, the description information of the fault root cause, and the fault solution for solving the fault. And, based on the obtained alarm correlation analysis result, not only can the fault root cause be quickly located and the fault handling time be shortened, but also the repeated and similar alarms can be aggregated by using the correlation relationship, so as to achieve the purpose of reducing alarm noise; further, based on the obtained alarm correlation analysis result, data support can also be provided for the intelligent operation and maintenance platform, so as to realize automatic fault handling; at the same time, potential security risks can be discovered by analyzing the alarm correlation relationship; furthermore, the repair operation can be automatically executed in combination with the root cause analysis result and the knowledge base; and, the possible abnormal behavior of the system can also be predicted by analyzing the alarm trend.

[0114] Further, after obtaining the alarm correlation analysis result, a correlation chart and / or a correlation report can also be generated according to the alarm correlation analysis result, and the correlation chart and / or the correlation report are displayed, so that the operation and maintenance personnel can quickly locate the fault root cause according to the displayed correlation chart and / or correlation report, and take corresponding treatment measures to achieve the purpose of improving the operation and maintenance efficiency, reducing the fault handling time and ensuring the business continuity; wherein, the obtained correlation chart and / or correlation report can be referred to Figure 6 as shown.

[0115] So far, the processing method of log alarm data recorded in the exemplary embodiments of the present disclosure has been fully implemented. Based on the foregoing content, it can be known that the processing method of log alarm data recorded in the exemplary embodiments of the present disclosure, on the one hand, solves many problems existing in traditional log alarm systems by introducing LLM and Multi-Agent technologies. For example, by using the natural language understanding and reasoning capabilities of LLM, it can accurately understand alarm information, identify complex relationships between alarms, thereby reducing alarm noise and improving the accuracy of alarms. For another example, by adopting the Multi-Agent collaborative working mode, parallel processing of multiple data sources can be realized, improving the overall processing efficiency and modularity of the system, and supporting more flexible expansion. On the other hand, based on multi-dimensional correlation analysis, it can help users quickly locate the root cause of faults, so as to reduce the fault handling time, improve the operation and maintenance efficiency, and ensure business continuity. On the third hand, the Action ability, independent alarm ability, and processing module execution ability of the alarm analysis Agent ensure the efficiency, real-time performance, and flexibility of alarm processing, improving the adaptive ability of the system. At the same time, the ability of the alarm analysis Agent to directly read data sources simplifies the data processing process and further improves the system efficiency.

[0116] That is to say, the processing method of log alarm data recorded in the exemplary embodiments of the present disclosure effectively solves the problems that traditional alarm systems cannot handle in complex environments, and can also achieve more intelligent, efficient, real-time, flexible, autonomous, and reliable alarm analysis and processing.

[0117] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the embodiment of the device of the present disclosure, please refer to the method embodiment of the present disclosure.

[0118] The exemplary embodiments of the present disclosure also provide a device for processing log alarm data. Specifically, referring to Figure 7 as shown, the device for processing log alarm data may include an information extraction model determination module 710, an alarm key information determination module 720, a correlation analysis module 730, and an alarm root cause reasoning module 740. Among them:

[0119] The information extraction model determination module 710 can be used to determine the data source category of the original log alarm data and allocate a key information extraction model corresponding to the data source category to the original log alarm data;

[0120] The alarm key information determination module 720 can be used to perform semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category;

[0121] The association relationship analysis module 730 can be used to perform multi-dimensional association analysis on the alarm key information of different data source categories based on a preset association analysis model, and obtain the association relationship between the alarm key information of different data source categories;

[0122] The alarm root cause inference module 740 can be used to perform alarm root cause inference on the association relationship between the alarm key information based on a preset fault inference model, and obtain the alarm association analysis result.

[0123] In an exemplary embodiment of the present disclosure, determining the data source category of the original log alarm data includes: determining the data source category of the original log alarm data according to the data source channel of the original log alarm data; wherein, the data source category includes at least one of an application log category, a system log category, a monitoring metric category, and other data source categories; the other data source categories include at least one of a third-party service log category, a security log category, and an operation log category; the key information extraction model includes at least one of a first key information extraction model corresponding to the application log category, a second key information extraction model corresponding to the system log category, a third key information extraction model corresponding to the monitoring metric category, and a fourth key information extraction model corresponding to the other data source categories.

[0124] In an exemplary embodiment of the present disclosure, performing semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category includes: invoking an application service interface based on the first key information extraction model to obtain the original application log alarm data corresponding to the application log category, and performing semantic analysis on the original application log alarm data to obtain the first alarm key information in the original application log alarm data; invoking a system log collection tool based on the second key information extraction model to obtain the original system log alarm data corresponding to the system log category, and performing semantic analysis on the original system log alarm data to obtain the second alarm key information in the original system log alarm data; invoking a status monitoring tool based on the third key information extraction model to obtain the original monitoring metric alarm data corresponding to the monitoring metric category, and performing semantic analysis on the original monitoring metric alarm data to obtain the third alarm key information in the original monitoring metric alarm data; invoking other log collection tools based on the fourth key information extraction model to obtain the original other log data corresponding to the other data source categories, and performing speech analysis on the original other log data to obtain the fourth alarm key information in the original other log data.

[0125] In an exemplary embodiment of the present disclosure, the first key information extraction model includes a first embedding mapping layer, a first encoding layer, and a first mixture-of-experts model layer; wherein, semantic analysis is performed on the original application log alarm data based on the first key information extraction model to obtain first alarm key information in the original application log alarm data, including: generating first basic information to be predicted according to the program running status information and program running error information in the original application log alarm data, and generating first context information to be predicted according to the program running event information in the original application log alarm data and a preset first model prompt parameter; performing embedding mapping processing on the first basic information to be predicted based on the first embedding mapping layer to obtain first log features, and performing embedding mapping processing on the first context information to be predicted based on the first embedding mapping layer to obtain a first context flag sequence; performing encoding processing on the first log features and the first context flag sequence based on the first encoding layer to obtain a first overall context representation, and performing semantic analysis on the first context flag sequence and the first overall context representation based on the first mixture-of-experts model layer to obtain first alarm key information in the original application log alarm data.

[0126] In an exemplary embodiment of the present disclosure, the first mixture-of-experts model layer includes a first gating network model and multiple first expert neural network models; wherein, based on the first mixture-of-experts model layer, semantic analysis is performed on the first context flag sequence and the first context overall representation to obtain first alarm key information in the original application log alarm data, including: based on the first gating network model according to the first context flag sequence, determining a first model weight of the first expert neural network model in the alarm type dimension, a second model weight in the alarm level dimension, a third model weight in the alarm time dimension, and a fourth model weight in the dimension of components involved in the alarm; based on the first model weight, the second model weight, the third model weight, and the fourth model weight, determining a first sub-goal neural network model required for performing semantic analysis tasks in the alarm type dimension, a second sub-goal neural network model required for performing semantic analysis tasks in the alarm level dimension, a third sub-goal neural network model required for performing semantic analysis tasks in the alarm time dimension, and a fourth sub-goal neural network model required for performing semantic analysis tasks in the dimension of components involved in the alarm from multiple first expert neural network models; inputting the first context flag sequence and the first context overall representation into the first target neural network model, the second target neural network model, the third target neural network model, and the fourth target neural network model respectively to obtain a first semantic prediction result in the alarm type dimension, a second semantic prediction result in the alarm level dimension, a third semantic prediction result in the alarm time dimension, and a fourth semantic prediction result in the dimension of components involved in the alarm; and generating first alarm key information in the original application log alarm data according to the first semantic prediction result, the second semantic prediction result, the third semantic prediction result, and the fourth semantic prediction result.

[0127] In an exemplary embodiment of the present disclosure, the processing device for the log alarm data further includes:

[0128] An alarm priority determination module, which can be used to determine the alarm category corresponding to the original application log alarm data according to the first alarm key information, and determine the alarm priority corresponding to the original application log alarm data according to the alarm category;

[0129] A component routing path determination module, which can be used to determine the component routing path according to the fourth semantic prediction result in the dimension of components involved in the alarm included in the first alarm key information if the alarm priority reaches a preset priority condition;

[0130] An operation instruction execution module, which can be used to determine an operation instruction corresponding to the alarm category based on the component routing path and execute the operation instruction to solve the abnormal behavior that generates the original application log alarm data.

[0131] In an exemplary embodiment of the present disclosure, the preset association analysis model includes a second embedding mapping layer, a second encoding layer, and a second mixture-of-experts model layer; wherein, performing multi-dimensional association analysis on the alarm key information of different data source categories based on the preset association analysis model to obtain the association relationship between the alarm key information of different data source categories includes: generating the basic key information to be predicted according to the alarm key information of different data source categories, and generating the context key information to be predicted according to the preset second model hint parameter; performing embedding mapping processing on the basic key information to be predicted based on the second embedding mapping layer to obtain key information features, and performing embedding mapping processing on the context key information to be predicted based on the second embedding mapping layer to obtain a key context flag sequence; performing encoding processing on the key information features and the key context flag sequence based on the second encoding layer to obtain an overall key context representation, and performing multi-dimensional association analysis on the key context flag sequence and the overall key context representation based on the second mixture-of-experts model layer to obtain the association relationship between the alarm key information of different data source categories.

[0132] In an exemplary embodiment of the present disclosure, the second mixture-of-experts model layer includes a second gated network model and multiple second expert neural network models; wherein, performing multi-dimensional association analysis on the key context flag sequence and the overall key context representation based on the second mixture-of-experts model layer to obtain the association relationship between the alarm key information of different data source categories includes: based on the second gated network model, determining the fifth model weight in the time relationship dimension, the sixth model weight in the space relationship dimension, and the seventh model weight in the causal relationship dimension of the second expert neural network model according to the key context flag sequence; determining the fifth sub-goal neural network model required for performing the association analysis task in the time relationship dimension, the sixth sub-goal neural network model required for performing the association analysis task in the space relationship dimension, and the seventh sub-goal neural network model required for performing the association analysis task in the causal relationship dimension from multiple second expert neural network models based on the fifth model weight, the sixth model weight, and the seventh model weight; inputting the key context flag sequence and the overall key context representation into the fifth target neural network model, the sixth target neural network model, and the seventh target neural network model respectively to obtain the first relationship prediction result in the time relationship, the second relationship prediction result in the space relationship dimension, and the third relationship prediction result in the causal relationship dimension; generating the association relationship between the alarm key information of different data source categories according to the first relationship prediction result, the second relationship prediction result, and the third relationship prediction result.

[0133] In an exemplary embodiment of the present disclosure, based on a preset fault reasoning model, the correlation relationship between key alarm information is used for alarm root cause reasoning to obtain an alarm correlation analysis result, including: generating correlation basic information to be predicted according to the correlation relationship between the key alarm information, and generating correlation context information to be predicted according to a preset third model prompt parameter; inputting the correlation basic information to be predicted and the correlation context information to be predicted into the preset fault reasoning model for alarm root cause reasoning to obtain an alarm correlation analysis result; wherein, the alarm correlation analysis result includes the time relationship, spatial relationship, causal relationship between alarms, the description information of the fault root cause, and the fault solution for solving the fault.

[0134] The specific details of each module in the above log alarm data processing device have been described in detail in the corresponding log alarm data processing method, so they will not be repeated here.

[0135] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0136] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be executed in this specific order, or that all the steps shown must be executed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0137] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is further provided. Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as a circuit, module, or system here.

[0138] Next, refer to Figure 8 to describe the electronic device 800 according to this embodiment of the present disclosure. Figure 8 The shown electronic device 800 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0139] AsFigure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one of the above-mentioned processing units 810, at least one of the above-mentioned storage units 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0140] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of this specification above. For example, the processing unit 810 can execute steps such as Figure 1 shown in step S110: determine the data source category of the original log alarm data, and assign a key information extraction model corresponding to the data source category to the original log alarm data; step S120: perform semantic analysis on the original log alarm data based on the key information extraction model to obtain the alarm key information in the original log alarm data corresponding to the data source category; step S130: perform multi-dimensional correlation analysis on the alarm key information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the alarm key information of different data source categories; step S140: perform alarm root cause reasoning on the correlation relationship between the alarm key information based on a preset fault reasoning model to obtain the alarm correlation analysis result.

[0141] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8201 and / or a cache storage unit 8202, and may further include a read-only storage unit (ROM) 8203.

[0142] The storage unit 820 may further include a program / utilities 8204 having a set (at least one) of program modules 8205. Such program modules 8205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0143] The bus 830 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0144] The electronic device 800 can also communicate with one or more external devices 900 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 850. Moreover, the electronic device 800 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 860. As shown in the figure, the network adapter 860 communicates with other modules of the electronic device 800 through the bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0145] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by the way of software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (for example, the computing device can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0146] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of this specification is stored. In some possible implementation manners, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0147] The program product for implementing the above method according to the embodiments of the present disclosure can adopt a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device.

[0148] The program product may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0150] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0151] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0152] In addition, the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0153] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention herein disclosed. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known or customary techniques in the art that are not invented by the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

1. A method for processing log alarm data, characterized in that: include: Determine the data source category of the original log alarm data, and assign a key information extraction model corresponding to the data source category to the original log alarm data; Performing semantic analysis on the original log alarm data based on the key information extraction model to obtain key alarm information in the original log alarm data corresponding to the data source category; Based on the preset correlation analysis model, a multi-dimensional correlation analysis is performed on the key alarm information of different data source categories to obtain the correlation relationship between the key alarm information of different data source categories; Based on the preset fault reasoning model, the root cause of the alarm is inferred from the correlation between the key alarm information to obtain the alarm correlation analysis result.

2. The method for processing log alarm data according to claim 1, characterized in that: Determine the data source category of the original log alarm data, including: Determine the data source category of the original log alarm data according to the data source channel of the original log alarm data; wherein the data source category includes at least one of the application log category, the system log category, the monitoring indicator category and other data source categories; the other data source categories include at least one of the third-party service log category, the security log category and the operation log category; The key information extraction model includes at least one of a first key information extraction model corresponding to the application log category, a second key information extraction model corresponding to the system log category, a third key information extraction model corresponding to the monitoring indicator category, and a fourth key information extraction model corresponding to other data source categories.

3. The method for processing log alarm data according to claim 2, characterized in that: Performing semantic analysis on the original log alarm data based on the key information extraction model to obtain key alarm information in the original log alarm data corresponding to the data source category, including: Based on the first key information extraction model, an application service interface is called to obtain original application log alarm data corresponding to the application log category, and semantic analysis is performed on the original application log alarm data to obtain first alarm key information in the original application log alarm data; Calling a system log collection tool based on the second key information extraction model to obtain original system log alarm data corresponding to the system log category, and performing semantic analysis on the original system log alarm data to obtain second alarm key information in the original system log alarm data; Based on the third key information extraction model, the status monitoring tool is called to obtain original monitoring indicator alarm data corresponding to the monitoring indicator category, and the original monitoring indicator alarm data is semantically analyzed to obtain the third alarm key information in the original monitoring indicator alarm data; Based on the fourth key information extraction model, other log collection tools are called to obtain original other log data corresponding to other data source categories, and voice analysis is performed on the original other log data to obtain the fourth alarm key information in the original other log data.

4. The method for processing log alarm data according to claim 3, characterized in that: The first key information extraction model includes a first embedding mapping layer, a first encoding layer and a first hybrid expert model layer; The original application log alarm data is semantically analyzed based on the first key information extraction model to obtain the first alarm key information in the original application log alarm data, including: Generate first basic information to be predicted according to program running state information and program running error information in the original application log alarm data, and generate first context information to be predicted according to program running event information in the original application log alarm data and preset first model prompt parameters; Performing embedding mapping processing on the first basic information to be predicted based on the first embedding mapping layer to obtain a first log feature, and performing embedding mapping processing on the first context information to be predicted based on the first embedding mapping layer to obtain a first context flag sequence; Based on the first coding layer, the first log feature and the first context flag sequence are encoded to obtain a first context overall representation, and based on the first hybrid expert model layer, the first context flag sequence and the first context overall representation are semantically analyzed to obtain the first alarm key information in the original application log alarm data.

5. The method for processing log alarm data according to claim 4, characterized in that: The first hybrid expert model layer includes a first gating network model and a plurality of first expert neural network models; The first context flag sequence and the first context overall representation are semantically analyzed based on the first hybrid expert model layer to obtain the first alarm key information in the original application log alarm data, including: Determine, based on the first gating network model and according to the first context flag sequence, a first model weight of the first expert neural network model in the dimension of alarm type, a second model weight in the dimension of alarm level, a third model weight in the dimension of alarm time, and a fourth model weight in the dimension of components involved in the alarm; Based on the first model weight, the second model weight, the third model weight and the fourth model weight, determine from a plurality of first expert neural network models a first sub-target neural network model required for performing a semantic analysis task on an alarm type dimension, a second sub-target neural network model required for performing a semantic analysis task on an alarm level dimension, a third sub-target neural network model required for performing a semantic analysis task on an alarm time dimension, and a fourth sub-target neural network model required for performing a semantic analysis task on a component dimension involved in an alarm; Inputting the first context flag sequence and the overall representation of the first and second contexts into the first target neural network model, the second target neural network model, the third target neural network model and the fourth target neural network model respectively, to obtain a first semantic prediction result in the dimension of alarm type, a second semantic prediction result in the dimension of alarm level, a third semantic prediction result in the dimension of alarm time and a fourth semantic prediction result in the dimension of components involved in the alarm; The first alarm key information in the original application log alarm data is generated according to the first semantic prediction result, the second semantic prediction result, the third semantic prediction result and the fourth semantic prediction result.

6. The method for processing log alarm data according to any one of claims 3 to 5, characterized in that: The method for processing log alarm data also includes: Determine the alarm category corresponding to the original application log alarm data according to the first alarm key information, and determine the alarm priority corresponding to the original application log alarm data according to the alarm category; If the alarm priority reaches a preset priority condition, determining a component routing path according to a fourth semantic prediction result on a component dimension involved in the alarm included in the first alarm key information; An operation instruction corresponding to the alarm category is determined based on the component routing path, and the operation instruction is executed to resolve the abnormal behavior that generates the original application log alarm data.

7. The method for processing log alarm data according to claim 1, characterized in that: The preset association analysis model includes a second embedding mapping layer, a second encoding layer, and a second hybrid expert model layer; Among them, based on the preset correlation analysis model, a multi-dimensional correlation analysis is performed on the alarm key information of different data source categories to obtain the correlation relationship between the alarm key information of different data source categories, including: Generate basic key information to be predicted according to the alarm key information of different data source categories, and generate context key information to be predicted according to the preset second model prompt parameters; Performing embedding mapping processing on the basic key information to be predicted based on the second embedding mapping layer to obtain key information features, and performing embedding mapping processing on the context key information to be predicted based on the second embedding mapping layer to obtain a key context flag sequence; Based on the second coding layer, the key information features and the key context flag sequence are encoded to obtain an overall representation of the key context, and based on the second hybrid expert model layer, a multi-dimensional correlation analysis is performed on the key context flag sequence and the overall representation of the key context to obtain the correlation relationship between the alarm key information of different data source categories.

8. The method for processing log alarm data according to claim 7, characterized in that: The second hybrid expert model layer includes a second gating network model and a plurality of second expert neural network models; Among them, based on the second hybrid expert model layer, a multi-dimensional correlation analysis is performed on the key context flag sequence and the key context overall representation to obtain the correlation relationship between the alarm key information of different data source categories, including: Determine, based on the second gating network model and according to the key context marker sequence, a fifth model weight in the time relationship dimension, a sixth model weight in the spatial relationship dimension, and a seventh model weight in the causal relationship dimension of the second expert neural network model; Based on the fifth model weight, the sixth model weight and the seventh model weight, determining from a plurality of second expert neural network models a fifth sub-target neural network model required for performing the association analysis task in the temporal relationship dimension, a sixth sub-target neural network model required for performing the association analysis task in the spatial relationship dimension, and a seventh sub-target neural network model required for performing the association analysis task in the causal relationship dimension; Inputting the key context marker sequence and the key context overall representation into the fifth target neural network model, the sixth target neural network model and the seventh target neural network model respectively, obtaining a first relationship prediction result in terms of time relationship, a second relationship prediction result in terms of spatial relationship dimension and a third relationship prediction result in terms of causal relationship dimension; Based on the first relationship prediction result, the second relationship prediction result and the third relationship prediction result, an association relationship between the alarm key information of different data source categories is generated.

9. The method for processing log alarm data according to claim 1, characterized in that: Based on the preset fault reasoning model, the root cause of the alarm is inferred based on the correlation between the key alarm information to obtain the alarm correlation analysis results, including: Generate the associated basic information to be predicted according to the associated relationship between the alarm key information, and generate the associated context information to be predicted according to the preset third model prompt parameter; Inputting the associated basic information to be predicted and the associated context information to be predicted into a preset fault reasoning model to perform alarm root cause reasoning to obtain an alarm correlation analysis result; The alarm correlation analysis result includes the temporal relationship, spatial relationship, causal relationship, description information of the root cause of the fault, and the fault solution for solving the fault.

10. A device for processing log alarm data, characterized in that: include: An information extraction model determination module is used to determine the data source category of the original log alarm data and assign a key information extraction model corresponding to the data source category to the original log alarm data; An alarm key information determination module is used to perform semantic analysis on the original log alarm data based on a key information extraction model to obtain alarm key information in the original log alarm data corresponding to the data source category; The correlation analysis module is used to perform multi-dimensional correlation analysis on the key alarm information of different data source categories based on a preset correlation analysis model to obtain the correlation relationship between the key alarm information of different data source categories; The alarm root cause reasoning module is used to perform alarm root cause reasoning on the correlation between alarm key information based on a preset fault reasoning model to obtain alarm correlation analysis results.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing log alarm data according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the log alarm data processing method according to any one of claims 1 to 9 by executing the executable instructions.

Images