An abnormal log analysis method and device, electronic equipment and storage medium

Abstract

The application discloses an abnormal log analysis method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an initial abnormal log data set; the initial abnormal log data set contains a plurality of initial abnormal log data; inputting each initial abnormal log data into a target multi-dimensional decision abnormal log analysis model, calculating a log abnormality coefficient corresponding to each initial abnormal log data; determining target abnormal log data in the initial abnormal log data set according to each log abnormality coefficient and a target log abnormality detection threshold; determining an abnormal business scenario corresponding to the target abnormal log data based on the target abnormal log data; and determining a target abnormal reason of the target abnormal log data according to the abnormal business scenario and the log abnormality coefficient of the target abnormal log data corresponding to the abnormal business scenario. The application can comprehensively analyze the root cause of abnormal problems, improve the efficiency and accuracy of abnormal log analysis, and can be widely applied in the technical field of data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an anomaly log analysis method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of information technology and the increasing complexity of enterprise operation and maintenance management, the analysis and processing of massive logs has become a challenge for operation and maintenance work. Faced with massive amounts of system and application logs, traditional log analysis methods are no longer sufficient to meet the requirements of real-time performance, accuracy, and automation.

[0003] In related technologies, the main methods for anomaly log analysis include: first, monitoring key indicators to detect log anomalies, followed by manual investigation of the logs to determine the cause of the anomalies; second, monitoring key business processes by embedding monitoring points to output anomaly logs for anomaly detection in key business processes; and third, analyzing and detecting anomalies by aggregating and monitoring abnormal errors in the logs and reviewing out-of-threshold dispatch orders. However, these methods suffer from problems such as incomplete analysis and low analysis efficiency.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of this application aim to at least partially solve one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose an anomaly log analysis method, apparatus, electronic device, and storage medium, capable of comprehensively analyzing the root causes of anomalies and improving the efficiency and accuracy of anomaly log analysis.

[0006] To achieve the above objectives, one aspect of this application proposes an anomaly log analysis method, the method comprising the following steps:

[0007] Obtain the initial exception log dataset; wherein, the initial exception log dataset contains several initial exception log data;

[0008] Each of the initial abnormal log data is input into the target multidimensional decision abnormal log analysis model to calculate the log abnormality coefficient corresponding to each of the initial abnormal log data.

[0009] Based on the log anomaly coefficients and the target log anomaly detection threshold, the target abnormal log data in the initial abnormal log dataset is determined;

[0010] Based on the target anomaly log data, determine the anomaly business scenario corresponding to the target anomaly log data;

[0011] The target anomaly cause of the target anomaly log data is determined based on the abnormal business scenario and the log anomaly coefficient of the target abnormal log data corresponding to the abnormal business scenario.

[0012] In some embodiments, determining the target abnormal log data in the initial abnormal log dataset based on each of the log anomaly coefficients and the target log anomaly detection threshold includes:

[0013] Based on each of the log anomaly coefficients and the target log anomaly detection threshold, determine whether several of the initial abnormal log data in the initial abnormal log dataset are identified as the target abnormal log data.

[0014] If the log anomaly coefficient corresponding to the initial abnormal log data is greater than the target log anomaly detection threshold, then the initial abnormal log data corresponding to the log anomaly coefficient that is greater than the target log anomaly detection threshold is determined as the target abnormal log data.

[0015] In some embodiments, determining the target anomaly cause of the target anomaly log data based on the abnormal business scenario and the log anomaly coefficient of the target abnormal log data corresponding to the abnormal business scenario includes:

[0016] If the number of abnormal business scenarios is single, then the abnormal business scenario is determined as the target abnormal cause of the target abnormal log data;

[0017] If there are several abnormal business scenarios, the log abnormality coefficients of the target abnormal log data corresponding to each abnormal business scenario are sorted in reverse order to obtain a sorting result, and the target abnormality cause of the target abnormal log data is determined based on the sorting result.

[0018] In some embodiments, inputting each of the initial abnormal log data into the target multidimensional decision-making abnormal log analysis model and calculating the log abnormality coefficient corresponding to each of the initial abnormal log data includes:

[0019] Each of the initial abnormal log data is input into the target multidimensional decision-making abnormal log analysis model, and the log abnormality coefficient corresponding to each of the initial abnormal log data is calculated using a multidimensional decision-making calculation formula, which is:

[0020]

[0021] Wherein, N represents the log anomaly coefficient, T represents the initial abnormal log data, n represents a natural number, Z represents the sensitivity weight business scenario corresponding to the target multidimensional decision-making abnormal log analysis model, i represents the i-th type of indicator, j represents the j-th evaluation value, Zij represents the sensitivity weight business scenario corresponding to the j-th evaluation value of the i-th type of indicator, X represents the sensitivity weight of the target multidimensional decision-making abnormal log analysis model, and Xij represents the sensitivity weight of the target multidimensional decision-making abnormal log analysis model corresponding to the j-th evaluation value of the i-th type of indicator.

[0022] In some embodiments, before inputting each of the initial abnormal log data into the target multidimensional decision-making abnormal log analysis model and calculating the log abnormality coefficient corresponding to each of the initial abnormal log data, the method further includes a step of pre-constructing the target multidimensional decision-making abnormal log analysis model, which specifically includes the following steps:

[0023] Get initial log data;

[0024] The initial log data is subjected to log detection processing to obtain log detection results;

[0025] The sensitivity of the log detection results in the current business scenario is calculated by using a multidimensional decision anomaly log analysis model to be trained, and several sensitivity weights are obtained based on the current business scenario; wherein, the multidimensional decision anomaly log analysis model to be trained consists of several evaluation models;

[0026] Based on the constraints of the target scenario, several different types of business scenarios are respectively taken as the current business scenario. The process of performing sensitivity calculation on the log detection results of the current business scenario through the multi-dimensional decision anomaly log analysis model to be trained, and obtaining several sensitivity weights based on the current business scenario, is repeated until several sensitivity weights corresponding to several current business scenarios under the constraints of the target scenario are obtained.

[0027] Based on several sensitivity weights corresponding to the current business scenarios, the target multidimensional decision anomaly log analysis model is obtained by configuring corrected weights on the multidimensional decision anomaly log analysis model to be trained.

[0028] In some embodiments, the step of performing log detection processing on the initial log data to obtain log detection results includes:

[0029] The initial log data is preprocessed to obtain the first serialized data;

[0030] Principal component analysis is performed on the first serialized data to obtain the target feature values;

[0031] The target feature vector is determined based on the target feature value, and a projection matrix is ​​constructed based on the target feature vector;

[0032] The target projection threshold is determined based on the first serialized data and the projection matrix;

[0033] Obtain the second serialized data;

[0034] The second serialized data is projected onto the projection matrix, and the log detection result is obtained based on the projection value corresponding to the projection point of the second serialized data in the projection matrix and the target projection threshold.

[0035] In some embodiments, the sensitivity calculation of the log detection results of the current business scenario using the multidimensional decision anomaly log analysis model to be trained, to obtain several sensitivity weights based on the current business scenario, includes:

[0036] In the current business scenario, the steps of performing log detection processing on the initial log data and obtaining log detection results are returned by several evaluation models in the multidimensional decision anomaly log analysis model to be trained, until several sensitivity weights based on the current business scenario are obtained.

[0037] To achieve the above objectives, another aspect of this application proposes an anomaly log analysis device, which includes the following modules:

[0038] The data acquisition module is used to acquire an initial exception log dataset; wherein, the initial exception log dataset contains several initial exception log data.

[0039] The log anomaly coefficient calculation module is used to input each of the initial abnormal log data into the target multidimensional decision anomaly log analysis model and calculate the log anomaly coefficient corresponding to each of the initial abnormal log data.

[0040] The target abnormal log determination module is used to determine the target abnormal log data in the initial abnormal log dataset based on each of the log abnormality coefficients and the target log abnormality detection threshold.

[0041] An abnormal business scenario determination module is used to determine the abnormal business scenario corresponding to the target abnormal log data based on the target abnormal log data.

[0042] The target anomaly cause determination module is used to determine the target anomaly cause of the target anomaly log data based on the anomaly business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the anomaly business scenario.

[0043] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0044] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0045] The embodiments of this application include at least the following beneficial effects: This application provides an anomaly log analysis method, apparatus, electronic device, and storage medium. This solution obtains an initial anomaly log dataset; wherein, the initial anomaly log dataset contains several initial anomaly log data; each initial anomaly log data is input into a target multidimensional decision anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data; based on each log anomaly coefficient and a target log anomaly detection threshold, target anomaly log data in the initial anomaly log dataset is determined; based on the target anomaly log data, the anomaly business scenario corresponding to the target anomaly log data is determined; based on the anomaly business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the anomaly business scenario, the target anomaly cause of the target anomaly log data is determined. This application first uses a target multidimensional decision-making anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data. This allows for a numerical assessment of the anomaly degree of each anomaly log data, providing a basis for subsequent anomaly log detection. Then, based on a set anomaly detection threshold, log data with a high degree of anomaly is filtered out. This allows attention to be focused on important anomalies, avoiding the processing of irrelevant data and improving the efficiency of anomaly log detection. Corresponding anomaly log data to specific anomaly business scenarios helps to more accurately locate and understand log anomalies. Finally, based on the anomaly business scenario and log anomaly coefficient, the root cause of the anomaly problem can be objectively and comprehensively analyzed, improving the efficiency and accuracy of anomaly log analysis. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the anomaly log analysis method based on data tracking provided in this application embodiment;

[0047] Figure 2 This is a flowchart illustrating the steps of an anomaly log analysis method provided in an embodiment of this application;

[0048] Figure 3 This is a flowchart illustrating the steps involved in constructing a target multidimensional decision-making anomaly log analysis model, as provided in an embodiment of this application.

[0049] Figure 4This is a flowchart illustrating the steps of log detection provided in an embodiment of this application;

[0050] Figure 5 This is a flowchart of the steps of principal component analysis provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of an anomaly log analysis method provided in an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the process for multi-model sensitivity measurement based on a single scenario provided in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the process for multi-model sensitivity measurement based on multiple scenarios provided in the embodiments of this application;

[0054] Figure 9 This is a schematic diagram of the structure of an anomaly log analysis device provided in an embodiment of this application;

[0055] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0057] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0058] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0061] 1) Long Short-Term Memory (LSTM): LSTM is a type of recurrent neural network designed to address the long-term dependency problem inherent in general RNNs (Recurrent Neural Networks). All RNNs have a chain-like structure of repeating neural network modules.

[0062] 2) Principal Component Analysis (PCA): Also known as principal component analysis, it aims to use the idea of ​​dimensionality reduction to transform multiple indicators into a few comprehensive indicators.

[0063] 3) Vector: In mathematics, a vector is a quantity that has magnitude and direction. It can be represented as an ordered array; in geometry, a vector is usually represented as a directed line segment with a specific start and end point.

[0064] Currently, as operators fully implement their cloud transformation and digital transformation strategies, in order to achieve the vision of "superior customer experience, ultra-fast product service, and intelligent cloud and network operation," they are gradually transforming system faults from rapid location to root cause analysis, and from manual operation and maintenance to automated and intelligent operation and maintenance. However, the current production situation is characterized by numerous applications and nodes, with insufficient automated and intelligent auxiliary means. Many operation and maintenance scenarios involve log retrieval and analysis of systems and applications, but the parsing of massive logs lacks detection methods, and potential problems cannot be exposed as early as possible, thus missing the best opportunity to solve problems.

[0065] As an example, relevant operators currently have several implementation plans for abnormal log analysis: First, they monitor key indicators to detect whether the logs are abnormal, and then manually check the relevant logs to determine the cause of the log abnormality; second, they conduct instrumentation monitoring through key business processes, and output abnormal logs through instrumentation to detect abnormal logs for key businesses; third, they conduct abnormal log analysis and detection by aggregating and monitoring abnormal errors in the logs and reviewing order dispatch exceeding the threshold.

[0066] Currently, in the process of platform log anomaly maintenance, relevant operators face two main challenges. Firstly, retrospectively locating the root cause of log problems after a failure occurs requires significant effort and results in lengthy recovery times for both the fault and the application system, impacting user experience. Secondly, there is a lack of real-time anomaly log detection methods, making it difficult to provide timely fault warnings even with substantial manpower. None of the aforementioned technologies effectively address the issues of root cause location and early fault identification while ensuring business and service continuity.

[0067] For example, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the anomaly log analysis method based on data tracking provided in this application embodiment; as follows: Figure 1 As shown, the implementation of the point-based anomaly log analysis method mainly includes four aspects: point-based implementation of key log information, fault report reception and log data collection, log data analysis, and handling of anomaly log issues. Specifically:

[0068] 1) Tracking of key log information: For key business processes and business components, track the status of the platform and application by setting tracking specifications.

[0069] 2) Fault report reception and log data collection: Fault report information on the business side is collected through multiple channels such as the service desk, and key business processes are collected through embedded logs;

[0070] 3) Log data analysis: This involves two aspects. On the one hand, for critical business processes that have been included in the management system, logs can be searched using keywords to identify abnormal situations and determine whether there are any anomalies in the logs. On the other hand, for logs that have not been included in the management system, the causes of anomalies can be obtained through manual analysis by operations and maintenance personnel.

[0071] 4) Handling abnormal log issues: Based on the results of log anomaly analysis, dispatch tasks to relevant personnel to handle platform or application system anomalies.

[0072] However, the above-mentioned method for analyzing abnormal logs based on data tracking faces the following problems when making judgments during abnormal log analysis and detection:

[0073] 1) Incomplete analysis: Relying on data tracking for key business processes limits the starting point to key business processes and fails to objectively and comprehensively analyze the root causes of abnormal problems.

[0074] 2) High cost and resource consumption: Expanding to other business scenarios requires business personnel to streamline processes and deploy tracking points, which consumes enterprise costs and is not cost-effective.

[0075] 3) Issues with the efficiency of anomaly log analysis: Analyzing anomalies through fault reports cannot quickly and promptly locate the cause of the anomaly, and it relies too much on the existing experience of operations and maintenance personnel, resulting in unsatisfactory problem localization.

[0076] In summary, the technical problems existing in the relevant technologies need to be improved.

[0077] In view of this, embodiments of this application provide an anomaly log analysis method, apparatus, electronic device, and storage medium. This solution involves: acquiring an initial anomaly log dataset; wherein the initial anomaly log dataset contains several initial anomaly log data; inputting each initial anomaly log data into a target multidimensional decision anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data; determining the target anomaly log data in the initial anomaly log dataset based on each log anomaly coefficient and the target log anomaly detection threshold; determining the anomaly business scenario corresponding to the target anomaly log data based on the target anomaly log data; and determining the target anomaly cause of the target anomaly log data based on the anomaly business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the anomaly business scenario. This application first uses a target multidimensional decision-making anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data. This allows for a numerical assessment of the anomaly degree of each anomaly log data, providing a basis for subsequent anomaly log detection. Then, based on a set anomaly detection threshold, log data with a high degree of anomaly is filtered out. This allows attention to be focused on important anomalies, avoiding the processing of irrelevant data and improving the efficiency of anomaly log detection. Corresponding anomaly log data to specific anomaly business scenarios helps to more accurately locate and understand log anomalies. Finally, based on the anomaly business scenario and log anomaly coefficient, the root cause of the anomaly problem can be objectively and comprehensively analyzed, improving the efficiency and accuracy of anomaly log analysis.

[0078] The abnormal log analysis method provided in this application relates to the field of data processing technology. The abnormal log analysis method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the abnormal log analysis method, but is not limited to the above forms.

[0079] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframes, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] Please see Figure 2 , Figure 2 This is an optional flowchart of an anomaly log analysis method provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, steps S201 to S205.

[0081] Step S201: Obtain the initial exception log dataset; wherein the initial exception log dataset contains several initial exception log data.

[0082] The initial anomaly log dataset contains multiple initial anomaly log data sets. These initial anomaly log data are log data that has not undergone detailed analysis and is initially considered to be abnormal. Furthermore, the initial anomaly log dataset typically includes various potential anomaly scenarios, which is beneficial for comprehensively identifying and handling various log anomaly issues.

[0083] In this embodiment of the application, by obtaining the initial abnormal log dataset, rich data support can be provided for subsequent abnormal log detection and analysis.

[0084] Step S202: Input each of the initial abnormal log data into the target multidimensional decision abnormal log analysis model, and calculate the log abnormality coefficient corresponding to each of the initial abnormal log data.

[0085] The log anomaly coefficient can be used to represent the degree of deviation between abnormal log data and normal behavior; that is, the log anomaly coefficient can be used to represent the degree of abnormality of the current abnormal log data.

[0086] In practice, by inputting each initial abnormal log data into the target multidimensional decision-making abnormal log analysis model, and using the multidimensional decision-making calculation formula to calculate the log abnormality coefficient corresponding to each initial abnormal log data, information from multiple dimensions can be comprehensively considered, thereby more accurately quantifying the degree of abnormality of each initial abnormal log data, making log abnormality detection more accurate.

[0087] In some embodiments, prior to step S202, the anomaly log analysis method may further include a step of pre-constructing a target multidimensional decision anomaly log analysis model. (See [link to relevant documentation]). Figure 3 , Figure 3 This is a flowchart illustrating the steps involved in constructing a target multidimensional decision-making anomaly log analysis model, as provided in an embodiment of this application. Figure 3 As shown, the pre-construction of the target multidimensional decision anomaly log analysis model may specifically include, but is not limited to, steps S301 to S305:

[0088] Step S301: Obtain initial log data;

[0089] The initial log data is unprocessed log data. It can be understood that the initial log data may be normal or abnormal. Further detection and processing are required to determine whether the initial log data is normal or abnormal.

[0090] The initial log data collection is a general log collection based on IT (Information Technology) systems. The main types of log data include four types: IaaS (Infrastructure as a Service) host system logs, PaaS (Platform as a Service) component system logs, SaaS (Software as a Service) application system logs, and cloud platform system logs (including application system logs).

[0091] For example, by subscribing to the IaaS host system logs, PaaS component system logs, SaaS application system logs, and the Kafka (distributed stream processing platform) topic where the application log data is located, the corresponding log data consumer groups can consume the data in real time, that is, the log data can be accessed in real time through data streams.

[0092] In practice, by acquiring initial log data, a data foundation is provided for subsequent log data detection and analysis. At the same time, the initial log data retains the originality and authenticity of the data, which is conducive to accurately identifying and analyzing abnormal situations.

[0093] Step S302: Perform log detection processing on the initial log data to obtain log detection results;

[0094] In practice, by performing log detection processing on the initial log data to obtain log detection results, abnormal log data can be quickly identified, providing direction for subsequent abnormal log data analysis.

[0095] In some specific embodiments, please refer to Figure 4 , Figure 4 This is a flowchart illustrating the steps of log detection provided in an embodiment of this application; as follows: Figure 4 As shown, step S302 may specifically include, but is not limited to, steps S401 to S406:

[0096] Step S401: Preprocess the initial log data to obtain the first serialized data;

[0097] Preprocessing can include data preprocessing steps such as removing missing values, handling outliers, and ensuring the consistency of timestamp formats.

[0098] For example, by removing missing values ​​and handling outliers, incomplete and erroneous data can be eliminated, reducing their interference with subsequent analysis and improving the accuracy and reliability of the analysis results. Ensuring consistency in timestamp formats helps to unify the time dimension of log data, facilitating time series analysis and other time-related analytical operations.

[0099] It should be noted that the data preprocessing example in this paper is only a simple example. In actual application scenarios, the required preprocessing method can be selected according to the actual situation. This application embodiment does not limit this.

[0100] In practice, by preprocessing the initial log data to obtain the first serialized data, the log data can be cleaned and standardized, improving data quality for subsequent data processing and feature extraction.

[0101] For example, taking a memory error detection scenario, to analyze log error issues, the serialized data can be represented as: time, memory usage ratio, memory error count, and whether a memory error occurred. The serialized data can be labeled as: Ai = {ai1, ai2}, where ai1 represents memory usage and ai2 represents memory error count.

[0102] Step S402: Perform principal component analysis on the first serialized data to obtain target feature values;

[0103] Please see Figure 5 , Figure 5 This is a flowchart illustrating the steps of principal component analysis provided in an embodiment of this application; as shown... Figure 5 As shown, step S402 may specifically include, but is not limited to, steps S501 to S504:

[0104] Step S501: The serialized data is centralized to obtain the target centralized data;

[0105] Specifically, principal component analysis can be used to center serialized data, which involves subtracting the mean from the data. The formula for centering is as follows:

[0106]

[0107] Among them, a ij Let represent the j-th evaluation value of the i-th indicator, where i represents the j-th indicator, j represents the j-th evaluation value, and n represents a natural number.

[0108] In practice, by centering the serialized data to obtain target-centered data, the mean of the data can be subtracted to eliminate scale differences between different features. Furthermore, the centered data (target-centered data) makes subsequent calculations and analyses more stable.

[0109] Step S502: Calculate the covariance matrix based on the target centered data;

[0110] The covariance matrix is ​​an important statistic in data analysis, reflecting the correlation between different features. Covariance is a fundamental concept in probability theory and mathematical statistics, characterizing the degree of correlation between multiple sets (columns) of data. The covariance matrix can be determined based on step a in step S501. ij This can be generated, with i (representing i types of indicators) as the number of rows in the matrix and j (representing j evaluation values ​​for the indicator) as the number of columns in the matrix.

[0111] In practice, by calculating the covariance matrix of the target centered data, the correlation between each feature can be obtained, providing important information for subsequent eigenvalue decomposition and feature selection.

[0112] Step S503: Perform eigenvalue decomposition on the covariance matrix to obtain the original eigenvalues ​​and the original eigenvectors corresponding to the original eigenvalues;

[0113] The original eigenvalues ​​and their corresponding original eigenvectors are obtained through eigenvalue decomposition.

[0114] Specifically, the covariance matrix is ​​decomposed into eigenvalues ​​to obtain the original eigenvalues ​​and their corresponding original eigenvectors, and the original eigenvalues ​​are then calculated. The formula for calculating the covariance value can be expressed as:

[0115]

[0116] Among them, H ij Let A represent the covariance calculated for i indicators and j evaluation values, where i represents the i-th type of indicator, j represents the j-th evaluation value, n represents a natural number, and A represents the variance calculated from a. ij The resulting matrix, A i It represents the full dataset of anomaly indicators, and T represents the transpose of the matrix.

[0117] In practical implementation, principal component analysis involves eigenvalue decomposition of the covariance matrix to obtain the main feature directions of the data (i.e., the original eigenvectors) and their corresponding importance (i.e., the original eigenvalues). The acquisition of the original eigenvalues ​​and original eigenvectors provides a basis for the subsequent retention of target eigenvalues.

[0118] Step S504: Determine the target feature value based on the magnitude relationship of the original feature values.

[0119] Specifically, the number of principal components to retain, z (i.e., the target number of eigenvalues), can be selected. For example, the required eigenvalues ​​(principal components) can be retained by analyzing the magnitude of the eigenvalues, such as retaining 90% of the variance. By determining the target eigenvalues, the dimensionality and complexity of the data can be reduced, while retaining the main information in the data, thereby improving the accuracy of subsequent analysis and prediction.

[0120] In the specific implementation, by performing principal component analysis on the first serialized data, the target feature value is obtained. This can retain the main information in the data while reducing the dimensionality and complexity of the data, which is convenient for subsequent feature analysis and application. The target feature value represents the key features of the data and can reflect the hidden patterns and relationships in the log data. It has important application value for log anomaly detection and fault diagnosis.

[0121] Step S403: Determine the target feature vector based on the target feature value, and construct a projection matrix based on the target feature vector;

[0122] Specifically, based on the selected z number of feature values, the first z feature vectors corresponding to the z number of feature values ​​are selected to form the projection matrix W. k = [v1, v2, v3, ... vn].

[0123] In practice, the projection matrix constructed based on the target feature vector can be used to project new log data (newly serialized data after preprocessing) into a space composed of the target key features, so as to perform log anomaly detection and fault analysis.

[0124] Step S404: Determine the target projection threshold based on the first serialized data and the projection matrix;

[0125] Specifically, the first serialized data is projected onto the projection matrix W. k The above can be used to determine the target projection threshold k. By setting the target projection threshold k, it can be used to detect whether the log data belongs to abnormal data.

[0126] In practice, the target projection threshold calculated based on the first serialized data and the projection matrix can be used to determine whether new log data is abnormal, that is, to identify important features in the dimensionality-reduced data, thereby improving the efficiency of data analysis.

[0127] Step S405: Obtain the second serialized data;

[0128] The second serialized data is the data obtained after preprocessing the reacquired initial log data, which is used to compare with the target projection threshold to preliminarily determine whether the log data is abnormal.

[0129] Step S406: Project the second serialized data onto the projection matrix, and obtain the log detection result based on the projection value corresponding to the projection point of the second serialized data in the projection matrix and the target projection threshold.

[0130] Specifically, the subsequent log data (second serialized data) is projected onto the projection matrix, and the projection value corresponding to the projection point of the second serialized data in the projection matrix is ​​compared with the target projection threshold. If the projection value corresponding to the projection point of the second serialized data in the projection matrix exceeds the target projection threshold, it can be determined that the log data has an error.

[0131] Step S303: The sensitivity of the log detection results of the current business scenario is calculated by the multidimensional decision anomaly log analysis model to be trained, and several sensitivity weights based on the current business scenario are obtained; wherein, the multidimensional decision anomaly log analysis model to be trained is composed of several evaluation models;

[0132] In some specific embodiments, step S303 may include: in the current business scenario, performing log detection processing on the initial log data and obtaining log detection results by returning several evaluation models in the multidimensional decision anomaly log analysis model to be trained, until several sensitivity weights based on the current business scenario are obtained.

[0133] The multidimensional decision anomaly log analysis model to be trained consists of several evaluation models, which may include, but are not limited to: Long Short-Term Memory Network (LSTM), PCA (Principal Component Analysis), and Convolutional Neural Network (CNN).

[0134] For the current business scenario, in step S303, the condition constraint of the current business scenario is a business scenario. For example, for the same scenario Z, different evaluation models are used, and step S302 is repeatedly executed to complete the model sensitivity test of scenario Z under different model scenarios (Xn). Multiple model sensitivities (weights) corresponding to scenario Z can be obtained, which can be labeled as Z = [z1, z2, z3 ... zn].

[0135] In the specific implementation, sensitivity measurement provides weight configuration for the multidimensional decision-making anomaly log analysis model based on the current business scenario, enabling the model to better adapt to and understand the data characteristics of specific business scenarios, thereby improving the applicability and accuracy of the model.

[0136] Step S304: Based on the target scenario constraints, several different types of business scenarios are respectively taken as the current business scenario. The step of performing sensitivity calculation on the log detection results of the current business scenario through the multi-dimensional decision anomaly log analysis model to be trained, and obtaining several sensitivity weights based on the current business scenario, is repeated until several sensitivity weights corresponding to several current business scenarios under the target scenario constraints are obtained.

[0137] In step S304, the target scenario condition constraint can be understood as the sensitivity weight calculation of different types of evaluation models based on different business scenarios.

[0138] In specific implementation, based on the known scenario log anomaly detection capability (reflected by model sensitivity), the sensitivity library of the detection model is enriched through scenario self-learning. Specifically, according to step S303, multiple model sensitivity weights for the same scenario Z can be obtained. In this embodiment, scenario self-learning can be implemented for different scenarios. By inputting scenario constraints under certain conditions and repeatedly executing step S303, several sensitivity weights Zi = [zi1, zi2, zi3...zin] for different scenarios can be obtained. Through multiple iterations and sensitivity weight calculations for different business scenarios, the model can comprehensively understand the characteristics of various business scenarios, enhancing its generalization ability and enabling it to effectively identify abnormal log data in different business scenarios.

[0139] Step S305: Configure corrected weights on the multidimensional decision anomaly log analysis model to be trained according to the sensitivity weights corresponding to the current business scenarios, and obtain the target multidimensional decision anomaly log analysis model.

[0140] In practical implementation, adjustment weights Xi = [xi1, xi2, xi3...xin] are set based on the production application effect (reflected by the model's sensitivity weights) for the application of different models (Xn), in order to enhance the decision-making effect in multi-dimensional decision-making. By adjusting the model according to the sensitivity weights, the model can be made more in line with the needs of actual business scenarios, improving the model's accuracy and reliability. At the same time, by optimizing model parameters and configurations, the false alarm rate can be reduced, and unnecessary resource waste and manual intervention can be minimized.

[0141] In some embodiments, step S202 may include: inputting each initial abnormal log data into the target multidimensional decision-making abnormal log analysis model, and calculating the log abnormality coefficient corresponding to each initial abnormal log data using a multidimensional decision-making calculation formula, wherein the multidimensional decision-making calculation formula is:

[0142]

[0143] Wherein, N represents the log anomaly coefficient, T represents the initial abnormal log data, n represents a natural number, Z represents the sensitivity weight business scenario corresponding to the target multidimensional decision-making abnormal log analysis model, i represents the i-th type of indicator, j represents the j-th evaluation value, Zij represents the sensitivity weight business scenario corresponding to the j-th evaluation value of the i-th type of indicator, X represents the sensitivity weight of the target multidimensional decision-making abnormal log analysis model, and Xij represents the sensitivity weight of the target multidimensional decision-making abnormal log analysis model corresponding to the j-th evaluation value of the i-th type of indicator.

[0144] In this embodiment of the application, by inputting each initial abnormal log data into the target multidimensional decision abnormal log analysis model, the log abnormality coefficient corresponding to each initial abnormal log data is calculated, and the abnormality degree of each abnormal log is evaluated in a numerical way, providing an objective basis for subsequent screening and analysis.

[0145] Step S203: Determine the target abnormal log data in the initial abnormal log dataset based on each of the log abnormality coefficients and the target log abnormality detection threshold;

[0146] In some embodiments, step S203 may include: determining whether a number of initial abnormal log data in the initial abnormal log dataset are target abnormal log data based on each log abnormal coefficient and the target log abnormal detection threshold; if the log abnormal coefficient corresponding to the initial abnormal log data is greater than the target log abnormal detection threshold, then the initial abnormal log data corresponding to the log abnormal coefficient that is greater than the target log abnormal detection threshold is determined as the target abnormal log data.

[0147] Specifically, by setting a target log anomaly detection threshold, log data with anomalies exceeding the threshold can be filtered out. The filtered log data can be considered as target abnormal log data, which serves as the basis for subsequent in-depth analysis.

[0148] In the specific implementation, based on the anomaly coefficients of each log and the target log anomaly detection threshold, it is determined whether several initial abnormal log data in the initial abnormal log dataset are identified as target abnormal log data. If the log anomaly coefficient corresponding to the initial abnormal log data is greater than the target log anomaly detection threshold, then the initial abnormal log data corresponding to the log anomaly coefficient greater than the target log anomaly detection threshold is identified as target abnormal log data. Based on the set anomaly detection threshold, log data with a higher degree of anomaly can be selected as the focus of subsequent analysis.

[0149] Step S204: Based on the target abnormal log data, determine the abnormal business scenario corresponding to the target abnormal log data;

[0150] For example, firstly, a log anomaly detection threshold q is set. If the log anomaly coefficient n is greater than the log anomaly detection threshold q, it means that the log anomaly can be detected by the multi-dimensional decision anomaly log analysis model, and the corresponding scenario i is the cause of the log anomaly.

[0151] In this embodiment of the application, based on the target abnormal log data, the abnormal business scenario corresponding to the target abnormal log data is determined. This allows the abnormal log data to be correlated with the specific abnormal business scenario, enabling more accurate location and judgment of log abnormalities.

[0152] Step S205: Determine the target anomaly cause of the target anomaly log data based on the abnormal business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the abnormal business scenario.

[0153] In some embodiments, step S205 may include: if the number of abnormal business scenarios is single, then the abnormal business scenario is determined as the target abnormal cause of the target abnormal log data; if the number of abnormal business scenarios is several, then the log abnormal coefficients of the target abnormal log data corresponding to each abnormal business scenario are sorted in reverse order to obtain a sorting result, and the target abnormal cause of the target abnormal log data is determined according to the sorting result.

[0154] In this embodiment, the target cause of the anomaly in the target abnormal log data is determined based on the abnormal business scenario and the log anomaly coefficient of the target abnormal log data corresponding to the abnormal business scenario. By combining the degree of anomaly in the abnormal log data with the abnormal business scenario for analysis, a deeper understanding of the root cause of the anomaly can be achieved, thereby enabling the implementation of corresponding solutions.

[0155] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0156] Please see Figure 6 , Figure 6 This is a schematic flowchart of an anomaly log analysis method provided in an embodiment of this application; as shown... Figure 6 As shown, the overall process of anomaly log analysis can include: collecting general logs from IT systems, building a multi-dimensional anomaly log analysis model, enriching the sensitivity library of the detection model through scenario self-learning, and using multi-dimensional decision algorithms to predict log analysis results.

[0157] Specifically, please refer to Figure 7 , Figure 7 This is a schematic diagram of the process for multi-model sensitivity calculation based on a single scenario, provided in an embodiment of this application; as follows: Figure 7 As shown, Figure 7 It can include the steps of general log collection from IT systems and building analysis models for mostly abnormal logs. Specifically, it can include... Figure 7 The steps are further subdivided into steps 101 to 108 as described below:

[0158] Step 101: General log collection and data preprocessing based on IT systems.

[0159] Specifically, log data is accessed in real time via data stream, and data preprocessing steps such as removing missing values, handling outliers, and ensuring the consistency of timestamp formats are completed to obtain serialized data.

[0160] Step 102: Use principal component analysis to center the serialized data, that is, subtract the mean from the serialized data.

[0161] Step 103: Construct a covariance matrix based on the centered data, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and their corresponding eigenvectors.

[0162] Step 104: Solve for the eigenvalues ​​and select the number z of principal components (eigenvalues) to be retained, i.e., retain z target eigenvalues. Specifically, the target eigenvalues ​​can be retained by analyzing their magnitudes, such as retaining 90% of the variance.

[0163] Step 105: Select the first z eigenvectors corresponding to the z eigenvalues ​​to form the projection matrix.

[0164] Step 106: Project the serialized data obtained after preprocessing in step 101 onto the projection matrix set in step 105. Based on the projection of the serialized data onto the projection matrix, a corresponding projection threshold k can be set. The projection threshold k is used to detect whether the log data is abnormal.

[0165] Step 107: Project the subsequent log data (new preprocessed serialized data) onto the projection matrix set in step 105, and compare it with the projection threshold k set in step 106. If the projection value corresponding to the projection point of the serialized data exceeds the projection threshold k, the serialized data can be determined to be log error data, i.e., abnormal log data.

[0166] Step 108: For the same scenario Z, use different evaluation models to calculate sensitivity. That is, repeat steps 102 to 107 until each evaluation model has completed step 107, so as to complete the model sensitivity test of the same scenario Z under different model scenarios (Xn) and obtain multiple model sensitivities (weights) corresponding to the same scenario Z.

[0167] Steps 101 to 108 enable real-time access to log data and the construction of a multi-dimensional abnormal log analysis model, thereby achieving rapid detection of anomalies within logs and sensitivity detection of multiple models for the same scenario.

[0168] To optimize the multidimensional anomaly log analysis model constructed in step 108 above, the sensitivity library of the detection model can be enriched through scenario self-learning, and adjustment weights can be set for the effects of different model applications based on production application results, thereby improving the decision-making effect in multidimensional decision-making. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of the process for multi-model sensitivity measurement based on multiple scenarios provided in the embodiments of this application; as follows: Figure 8 As shown, the process of predicting log analysis results can include enriching the sensitivity library of the detection model through scene self-learning and multi-dimensional decision algorithms. Specifically, it can include... Figure 8 The steps in the process are further subdivided into steps 201 to 206 as described below:

[0169] Step 201: According to step 108 above, multiple model sensitivity weights can be obtained in the same scenario Z. Furthermore, in this embodiment of the application, scenario self-learning can be achieved for different scenarios. That is, according to certain conditions, scenario constraints are input, and step 108 above is repeatedly executed to obtain multiple model sensitivity weights corresponding to different scenarios, so as to enrich the sensitivity library of the detection model.

[0170] Step 202: Based on the production application effect (reflected by the model sensitivity weight), set correction weights for the effects of different model applications to increase the decision-making effect in multidimensional decision-making. Based on this, the optimized target multidimensional decision-making anomaly log analysis model can be obtained.

[0171] Step 203: Input the abnormal log data into the optimized target multidimensional decision-making abnormal log analysis model, and use the multidimensional decision-making algorithm to calculate the log abnormality coefficient corresponding to the abnormal log data.

[0172] Step 204: Set the log anomaly detection threshold q. If the log anomaly coefficient n is greater than the log anomaly detection threshold q, it means that the target multidimensional decision anomaly log analysis model can detect log anomalies. When a log anomaly is detected, the corresponding scenario i is the cause of the log anomaly.

[0173] Step 205: If multiple scenarios are detected as abnormal, the cause of the abnormal scenario can be output in reverse order according to the log abnormality coefficient corresponding to the abnormal log data of each scenario.

[0174] Step 206: Based on the evaluation of the production application effect, repeat steps 201 to 205 to further optimize the multidimensional decision analysis model.

[0175] Through steps 201 to 206, the ability to detect log anomalies based on known scenarios can be enhanced by enriching the sensitivity library of the detection model through scenario self-learning, and by combining multi-dimensional decision-making algorithms to predict log analysis results, thereby improving the accuracy of log anomaly detection.

[0176] In a specific implementation, an anomaly log analysis method in this application embodiment may correspond to a multi-dimensional decision-making anomaly log analysis device. This multi-dimensional decision-making anomaly log analysis device may include a data acquisition unit, a control processing unit, a prediction unit, and an output unit. Specifically:

[0177] 1) Collection Unit: Collects general logs from IT systems. The main types of logs include four types: IaaS host system logs, PaaS component system logs, SaaS application system logs, and cloud platform system logs (including application system logs). Specifically, log data is accessed in real time via data stream.

[0178] 2) Control and processing unit: Used for data normalization processing and construction of multi-dimensional decision-making anomaly log analysis model, to achieve rapid detection of anomalies in logs and sensitivity detection of various models for different scenarios.

[0179] 3) Prediction Unit: Enables scenario self-learning and self-training, uses multi-dimensional decision-making algorithms to predict log analysis results, and provides maintenance personnel with quick analysis of the causes of log anomalies.

[0180] 4) Output Unit: Outputs and pushes the log anomaly prediction results and anomaly scenarios generated in the prediction unit to the expected location.

[0181] In this embodiment, on the one hand, the timeliness of log anomaly detection is improved by constructing a multi-dimensional decision-making anomaly log analysis; on the other hand, it supports automatic learning and training of scenarios, and realizes the accuracy of log anomaly detection and root cause localization based on the multi-dimensional decision-making algorithm, thereby improving the efficiency of front-line IT maintenance personnel in detecting abnormal log data and improving the efficiency of fault handling. In other words, through multi-dimensional decision-making anomaly log analysis, anomalies in the system can be quickly detected and located, enabling rapid localization of log anomaly problems and real-time detection of log anomalies, improving the efficiency of operation and maintenance work, and reducing operation and maintenance costs.

[0182] In summary, the abnormal log analysis method provided in this application can be applied to scenarios including but not limited to: timely discovery of problems or potential risks, root cause localization of log anomalies, and continuous optimization of application scenario development. Specifically:

[0183] 1) Timely detection of problems or potential risks: Real-time detection of platform and application anomaly logs enables timely detection of problems or potential risks.

[0184] 2) Trace the root cause of the problem: Locate the root cause of log anomalies.

[0185] 3) Application development scenario optimization: Anomalies that affect usage due to imperfections in development can prompt developers to make improvements.

[0186] It should be noted that this embodiment is only a brief illustrative description of the general process of the abnormal log analysis method. For detailed description of each step, please refer to the relevant content in the foregoing embodiments. It will not be repeated here. It is understood that the present invention does not limit this.

[0187] This application embodiment obtains an initial abnormal log dataset; wherein the initial abnormal log dataset contains several initial abnormal log data; each initial abnormal log data is input into a target multidimensional decision-making abnormal log analysis model to calculate the log abnormality coefficient corresponding to each initial abnormal log data; based on each log abnormality coefficient and the target log abnormality detection threshold, the target abnormal log data in the initial abnormal log dataset is determined; based on the target abnormal log data, the abnormal business scenario corresponding to the target abnormal log data is determined; based on the abnormal business scenario and the log abnormality coefficient of the target abnormal log data corresponding to the abnormal business scenario, the target abnormal cause of the target abnormal log data is determined. This application first uses a target multidimensional decision-making anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data. This allows for a numerical assessment of the anomaly degree of each anomaly log data, providing a basis for subsequent anomaly log detection. Then, based on a set anomaly detection threshold, log data with a high degree of anomaly is filtered out. This allows attention to be focused on important anomalies, avoiding the processing of irrelevant data and improving the efficiency of anomaly log detection. Corresponding anomaly log data to specific anomaly business scenarios helps to more accurately locate and understand log anomalies. Finally, based on the anomaly business scenario and log anomaly coefficient, the root cause of the anomaly problem can be objectively and comprehensively analyzed, improving the efficiency and accuracy of anomaly log analysis.

[0188] Please see Figure 9 This application also provides an anomaly log analysis device 900, which can implement the above-mentioned anomaly log analysis method. The device 900 includes the following modules:

[0189] The data acquisition module 901 is used to acquire an initial exception log dataset; wherein, the initial exception log dataset contains several initial exception log data.

[0190] The log anomaly coefficient calculation module 902 is used to input each of the initial abnormal log data into the target multidimensional decision anomaly log analysis model and calculate the log anomaly coefficient corresponding to each of the initial abnormal log data.

[0191] The target abnormal log determination module 903 is used to determine the target abnormal log data in the initial abnormal log dataset based on each of the log abnormal coefficients and the target log abnormal detection threshold.

[0192] The abnormal business scenario determination module 904 is used to determine the abnormal business scenario corresponding to the target abnormal log data based on the target abnormal log data.

[0193] The target anomaly cause determination module 905 is used to determine the target anomaly cause of the target anomaly log data based on the anomaly business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the anomaly business scenario.

[0194] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0195] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described exception log analysis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0196] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0197] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0198] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0199] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the exception log analysis method of the embodiments of this application.

[0200] Input / output interface 1003 is used to implement information input and output;

[0201] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0202] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0203] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0204] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described exception log analysis method.

[0205] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0206] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0207] The abnormal log analysis method, abnormal log analysis device, electronic device, and storage medium provided in this application embodiment acquire an initial abnormal log dataset, wherein the initial abnormal log dataset contains several initial abnormal log data; each initial abnormal log data is input into a target multidimensional decision-making abnormal log analysis model to calculate the log abnormality coefficient corresponding to each initial abnormal log data; based on each log abnormality coefficient and a target log abnormality detection threshold, target abnormal log data in the initial abnormal log dataset is determined; based on the target abnormal log data, the abnormal business scenario corresponding to the target abnormal log data is determined; and based on the abnormal business scenario and the log abnormality coefficient of the target abnormal log data corresponding to the abnormal business scenario, the target abnormal cause of the target abnormal log data is determined. This application first uses a target multidimensional decision-making anomaly log analysis model to calculate the log anomaly coefficient corresponding to each initial anomaly log data. This allows for a numerical assessment of the anomaly degree of each anomaly log data, providing a basis for subsequent anomaly log detection. Then, based on a set anomaly detection threshold, log data with a high degree of anomaly is filtered out. This allows attention to be focused on important anomalies, avoiding the processing of irrelevant data and improving the efficiency of anomaly log detection. Corresponding anomaly log data to specific anomaly business scenarios helps to more accurately locate and understand log anomalies. Finally, based on the anomaly business scenario and log anomaly coefficient, the root cause of the anomaly problem can be objectively and comprehensively analyzed, improving the efficiency and accuracy of anomaly log analysis.

[0208] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0209] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0212] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0213] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0215] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0217] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0218] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An anomaly log analysis method, characterized in that, The method includes the following steps: Obtain the initial exception log dataset; wherein, the initial exception log dataset contains several initial exception log data; Each of the initial abnormal log data is input into the target multidimensional decision abnormal log analysis model to calculate the log abnormality coefficient corresponding to each of the initial abnormal log data. Based on the log anomaly coefficients and the target log anomaly detection threshold, the target abnormal log data in the initial abnormal log dataset is determined; Based on the target anomaly log data, determine the anomaly business scenario corresponding to the target anomaly log data; Based on the abnormal business scenario and the log abnormality coefficient of the target abnormal log data corresponding to the abnormal business scenario, the target abnormality cause of the target abnormal log data is determined; Before inputting each of the initial abnormal log data into the target multidimensional decision-making abnormal log analysis model and calculating the log abnormality coefficient corresponding to each of the initial abnormal log data, the method further includes a step of pre-constructing the target multidimensional decision-making abnormal log analysis model, wherein the pre-construction of the target multidimensional decision-making abnormal log analysis model includes: Get initial log data; The initial log data is subjected to log detection processing to obtain log detection results; The sensitivity of the log detection results in the current business scenario is calculated by using a multidimensional decision anomaly log analysis model to be trained, and several sensitivity weights are obtained based on the current business scenario; wherein, the multidimensional decision anomaly log analysis model to be trained consists of several evaluation models; Based on the constraints of the target scenario, several different types of business scenarios are respectively taken as the current business scenario. The process of performing sensitivity calculation on the log detection results of the current business scenario through the multi-dimensional decision anomaly log analysis model to be trained, and obtaining several sensitivity weights based on the current business scenario, is repeated until several sensitivity weights corresponding to several current business scenarios under the constraints of the target scenario are obtained. Based on several sensitivity weights corresponding to the current business scenarios, the target multidimensional decision anomaly log analysis model is obtained by configuring corrected weights on the multidimensional decision anomaly log analysis model to be trained.

2. The method according to claim 1, characterized in that, The step of determining the target abnormal log data in the initial abnormal log dataset based on each of the log abnormality coefficients and the target log abnormality detection threshold includes: Based on each of the log anomaly coefficients and the target log anomaly detection threshold, determine whether several of the initial abnormal log data in the initial abnormal log dataset are identified as the target abnormal log data. If the log anomaly coefficient corresponding to the initial abnormal log data is greater than the target log anomaly detection threshold, then the initial abnormal log data corresponding to the log anomaly coefficient that is greater than the target log anomaly detection threshold is determined as the target abnormal log data.

3. The method according to claim 1, characterized in that, The step of determining the target anomaly cause of the target anomaly log data based on the abnormal business scenario and the log anomaly coefficient of the target abnormal log data corresponding to the abnormal business scenario includes: If the number of abnormal business scenarios is single, then the abnormal business scenario is determined as the target abnormal cause of the target abnormal log data; If there are several abnormal business scenarios, the log abnormality coefficients of the target abnormal log data corresponding to each abnormal business scenario are sorted in reverse order to obtain a sorting result, and the target abnormality cause of the target abnormal log data is determined based on the sorting result.

4. The method according to claim 1, characterized in that, The step of inputting each of the initial abnormal log data into the target multidimensional decision-making abnormal log analysis model and calculating the log abnormality coefficient corresponding to each of the initial abnormal log data includes: Each of the initial abnormal log data is input into the target multidimensional decision-making abnormal log analysis model, and the log abnormality coefficient corresponding to each of the initial abnormal log data is calculated using a multidimensional decision-making calculation formula, which is: ， in, This is represented as the log anomaly coefficient. This is represented as the initial exception log data. Represented as natural numbers, This represents the sensitivity weight business scenario corresponding to the target multidimensional decision anomaly log analysis model. Represented as the first Class indicators, Represented as the first One evaluation value, Represented as the first The first category of indicators The sensitivity weighting of each evaluation value corresponds to a business scenario. The sensitivity weights are represented by the target multidimensional decision anomaly log analysis model. Represented as the first The first category of indicators The sensitivity weight of the target multidimensional decision anomaly log analysis model corresponding to each evaluation value.

5. The method according to claim 1, characterized in that, The process of performing log detection processing on the initial log data to obtain log detection results includes: The initial log data is preprocessed to obtain the first serialized data; Principal component analysis is performed on the first serialized data to obtain the target feature values; The target feature vector is determined based on the target feature value, and a projection matrix is ​​constructed based on the target feature vector; The target projection threshold is determined based on the first serialized data and the projection matrix; Obtain the second serialized data; The second serialized data is projected onto the projection matrix, and the log detection result is obtained based on the projection value corresponding to the projection point of the second serialized data in the projection matrix and the target projection threshold.

6. The method according to claim 1, characterized in that, The sensitivity calculation of the log detection results for the current business scenario using the multidimensional decision anomaly log analysis model to be trained yields several sensitivity weights based on the current business scenario, including: In the current business scenario, the steps of performing log detection processing on the initial log data and obtaining log detection results are returned by several evaluation models in the multidimensional decision anomaly log analysis model to be trained, until several sensitivity weights based on the current business scenario are obtained.

7. An anomaly log analysis device, characterized in that, The device includes the following modules: The data acquisition module is used to acquire an initial exception log dataset; wherein, the initial exception log dataset contains several initial exception log data. The log anomaly coefficient calculation module is used to input each of the initial abnormal log data into the target multidimensional decision anomaly log analysis model and calculate the log anomaly coefficient corresponding to each of the initial abnormal log data. The target abnormal log determination module is used to determine the target abnormal log data in the initial abnormal log dataset based on each of the log abnormality coefficients and the target log abnormality detection threshold. An abnormal business scenario determination module is used to determine the abnormal business scenario corresponding to the target abnormal log data based on the target abnormal log data. The target anomaly cause determination module is used to determine the target anomaly cause of the target anomaly log data based on the anomaly business scenario and the log anomaly coefficient of the target anomaly log data corresponding to the anomaly business scenario. The device is also used for: Get initial log data; The initial log data is subjected to log detection processing to obtain log detection results; The sensitivity of the log detection results in the current business scenario is calculated by using a multidimensional decision anomaly log analysis model to be trained, and several sensitivity weights are obtained based on the current business scenario; wherein, the multidimensional decision anomaly log analysis model to be trained consists of several evaluation models; Based on the constraints of the target scenario, several different types of business scenarios are respectively taken as the current business scenario. The process of performing sensitivity calculation on the log detection results of the current business scenario through the multi-dimensional decision anomaly log analysis model to be trained, and obtaining several sensitivity weights based on the current business scenario, is repeated until several sensitivity weights corresponding to several current business scenarios under the constraints of the target scenario are obtained. Based on several sensitivity weights corresponding to the current business scenarios, the target multidimensional decision anomaly log analysis model is obtained by configuring corrected weights on the multidimensional decision anomaly log analysis model to be trained.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images