System anomaly detection method, device and non-volatile storage medium

Abstract

The application discloses a system anomaly detection method and device and a nonvolatile storage medium. The method comprises the following steps: obtaining log records and tracking trajectory data of a distributed system to be detected; performing normalization processing on the log records and the tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and the target template tracking trajectories are the same; processing the target template logs and the target template tracking trajectories through a target LSTM network to obtain target predicted log records and target predicted tracking trajectory data; determining a running state of the distributed system to be detected, and determining whether the system to be detected is in an abnormal working state according to the target predicted log records, the target predicted tracking trajectory data and the running state. The application solves the technical problem of low detection accuracy caused by using single data to detect system anomalies in the related art.

Description

Technical Field

[0001] This application relates to the field of detection, and more specifically, to a system anomaly detection method, apparatus, and non-volatile storage medium. Background Technology

[0002] In related technologies, anomaly detection in distributed systems typically only utilizes one type of data: program logs or distributed trace data. This approach of relying on a single data source results in low accuracy for anomaly detection in these technologies.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a system anomaly detection method, apparatus, and non-volatile storage medium to at least solve the technical problem of low detection accuracy caused by using single data for system anomaly detection in related technologies.

[0005] According to one aspect of the embodiments of this application, a system anomaly detection method is provided, comprising: acquiring log records and tracking trajectory data of a distributed system to be detected; performing normalization processing on the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and target template tracking trajectories are the same; processing the target template logs and target template tracking trajectories through a target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; determining the operating state of the distributed system to be detected, and determining whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, target prediction tracking trajectory data and operating state.

[0006] Optionally, the log records include log constants and log variables. The steps for normalizing the log records include: extracting log constants from the log records; establishing a log template dictionary based on the log constants; and obtaining a target template log based on the log template dictionary. The target template log consists of multiple target extraction words, which are extraction words determined from the log template dictionary based on the first index corresponding to the log record.

[0007] Optionally, the tracking trajectory data consists of multiple span data, wherein the span data is used to record the operation information of the distributed system under test when processing external requests, and the operation information includes at least one of the following: start time, end time, service name, and HTTP path; the steps for normalizing the tracking trajectory data include: adding a first preset span at the starting point of the tracking trajectory data, and adding a second preset span at the ending point of the tracking trajectory data, wherein the first preset span is used to indicate the starting point of the tracking trajectory data, and the second preset span is used to indicate the ending point of the tracking trajectory data; after adding the first preset span and the second preset span, a first template tracking trajectory is obtained using a second index and a trajectory template set, wherein the first template tracking trajectory includes multiple template spans determined from the trajectory template set according to the second index, and the second index is an index determined based on the tracking trajectory data; a second template tracking trajectory is obtained based on the tracking trajectory template dictionary and the first template tracking trajectory, wherein the second template tracking trajectory is a target template tracking trajectory, and the target template tracking trajectory consists of multiple target character sequences, and the target character sequences are character sequences extracted from the tracking trajectory template dictionary.

[0008] Optionally, the step of simultaneously processing the target template log and the target template tracking trajectory through the target LSTM network to obtain the predicted log record and the predicted tracking trajectory data includes: constructing a target prediction task model based on the target template log and the target template tracking trajectory, wherein the target prediction task model is used to determine the predicted log record and the predicted tracking trajectory data; constructing a target LSTM network based on the target prediction task model; and simultaneously processing the target template log and the target template tracking trajectory through the target LSTM network to obtain the predicted log record and the predicted tracking trajectory data.

[0009] Optionally, the target LSTM network includes a template embedding layer, a first LSTM cluster, a second LSTM cluster, and a probability output layer. The template embedding layer is used to convert the target template log into a target template log sequence and to convert the target template tracking trajectory into a target template tracking trajectory sequence. The first LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template log sequence. The second LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template tracking trajectory sequence. The outputs of the first and second LSTM clusters are concatenated and fed back to the first and second LSTM clusters through an additional linear layer. The probability output layer outputs prediction results, including possible predicted log records and their probabilities of occurrence, as well as possible predicted tracking trajectory data and their probabilities of occurrence.

[0010] Optionally, the probability output layer includes a first linear layer and a second linear layer, wherein the first linear layer is used to determine the predicted log record and the second linear layer is used to determine the predicted tracking trajectory data.

[0011] Optionally, the steps of constructing the target LSTM network include: constructing a target cost function and training the target LSTM network using the target cost function, wherein the output of the target cost function is the joint cross-entropy loss of the target prediction log record and the target prediction tracking trajectory data.

[0012] According to another aspect of the embodiments of this application, a system anomaly detection device is also provided, comprising: an input module for acquiring log records and tracking trajectory data of a distributed system to be detected; a standardization module for standardizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and target template tracking trajectories are the same; a first processing module for processing the target template logs and target template tracking trajectories through a target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; and a second processing module for determining the operating state of the distributed system to be detected, and determining whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, the target prediction tracking trajectory data, and the operating state.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute a system anomaly detection method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes a system anomaly detection method during runtime.

[0015] In this embodiment, the following methods are employed: acquiring log records and tracking trajectory data of the distributed system to be detected; normalizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and target template tracking trajectories are identical; processing the target template logs and target template tracking trajectories using a target LSTM network to obtain target predicted log records and target predicted tracking trajectory data; determining the operating state of the distributed system to be detected; and determining whether the distributed system to be detected is in an abnormal working state based on the target predicted log records, target predicted tracking trajectory data, and operating state. By using a target LSTM network to process the target template logs and target template tracking trajectories to obtain target predicted log records and target predicted tracking trajectory data, the method achieves the goal of comprehensively considering log records and tracking trajectory data to determine whether the system is abnormal, thereby achieving a high-accuracy technical effect for system anomaly detection, and thus solving the technical problem of low detection accuracy caused by using single data for system anomaly detection in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a schematic diagram of the structure of an optional computer terminal device according to an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating a system anomaly detection method according to an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the architecture of a bidirectional LSTM unit according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the architecture of a system anomaly detection process according to an embodiment of this application;

[0021] Figure 5 This is a flowchart illustrating a system anomaly detection process according to an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a system anomaly detection device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Currently, the increasing complexity of multi-layered IT infrastructure, including the Internet of Things (IoT), distributed processing frameworks, databases, and operating systems, generates massive amounts of monitoring data. This leads to increased maintenance overhead for the entire distributed system, particularly in accurately detecting abnormal states where the system deviates from expected behavior—a key aspect of anomaly detection. Current monitoring data in distributed systems primarily falls into three categories: metrics, application logs, and distributed trace data. Existing distributed system anomaly detection methods rely solely on parsing log data or distributed trace data, resulting in limited accuracy. Considering that log data preserves system-level information, while trace data describes microservice-level execution intentions and interactions between components, both providing different perspectives on the distributed system's operational status, achieving efficient integration of these two types of information to improve anomaly detection efficiency is crucial.

[0026] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0027] According to an embodiment of this application, a method embodiment for system anomaly detection is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a system anomaly detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the system anomaly detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned system anomaly detection method for the application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0032] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0033] Under the above operating environment, embodiments of this application provide a system anomaly detection method, such as... Figure 2 As shown, the method includes the following steps:

[0034] Step S202: Obtain the log records and tracking trajectory data of the distributed system to be detected;

[0035] Specifically, in order to make full use of the monitoring data generated in the distributed system to achieve high-accuracy system anomaly detection, this application embodiment will consider both log data and tracking trajectory data, thereby more accurately reflecting the working status of the distributed system and laying the foundation for achieving more accurate system anomaly detection.

[0036] Step S204: Standardize the log records and tracking trajectory data to obtain the target template log and the target template tracking trajectory, wherein the data format of the target template log and the target template tracking trajectory is the same;

[0037] It should be noted that the raw log data and trace data generated by the system describe different levels of the system's operational status. They differ in information granularity and functionality and need to be represented within the same framework for subsequent processing. Furthermore, both types of data contain various information about specific operations being performed. Since some information is a fragmented description of the operation, both types of data require appropriate filtering and representation.

[0038] In the technical solution provided in step S202, the log records include log constants and log variables. The steps for normalizing the log records include: extracting log constants from the log records; establishing a log template dictionary based on the log constants; and obtaining a target template log based on the log template dictionary. The target template log consists of multiple target extraction words, which are extraction words determined from the log template dictionary based on the first index corresponding to the log record.

[0039] Here, the first index, denoted by t, describes the t-th extracted word index corresponding to the vocabulary in a single text message sample log template. The log template dictionary contains elements that are non-empty proper subsets of the vocabulary, and different log template dictionaries may differ. The raw logs generated by the system contain various information about the specific operation being performed. Since some information is a sporadic description of the operation, the log data belonging to unstructured text messages is normalized using the log template dictionary to facilitate subsequent algorithm execution.

[0040] Specifically, the log data mainly consists of a series of unstructured text messages arranged in chronological order. Each text message is generated based on logging instructions in the software's source code, such as `printf()` and `log.info()`. Since logging is an integral part of the distributed system program, it can act as a proxy for the program's execution workflow, allowing inference of normal execution patterns. Logs consist of constants and variables, called log templates and log parameters. Because log parameters typically exhibit significant variability, introducing considerable noise, the log parsing process generally extracts the constant portion, creating a log template dictionary from a given set of logs. To standardize log representation, log templates are tokenized. This represents a vocabulary of all tags in the log; additionally, since log templates can have different numbers of tags, a special [item name] is added to unify the representation of log templates. <speclog>Tags are used to ensure that each log has the same number of tags, and the size of the log template is limited using the max_log_size parameter.

[0041]

[0042] in Based on the index Mapped to Extracted words.

[0043] As an optional implementation, the tracking trajectory data consists of multiple span data, wherein the span data is used to record operation information when the distributed system under test processes external requests, and the operation information includes at least one of the following: start time, end time, service name, and HTTP path; the steps of normalizing the tracking trajectory data include: adding a first preset span at the starting point of the tracking trajectory data, and adding a second preset span at the ending point of the tracking trajectory data, wherein the first preset span is used to indicate the starting point of the tracking trajectory data, and the second preset span is used to indicate the ending point of the tracking trajectory data; after adding the first preset span and the second preset span, a first template tracking trajectory is obtained using a second index and a trajectory template set, wherein the first template tracking trajectory includes multiple template spans determined from the trajectory template set according to the second index, and the second index is an index determined based on the tracking trajectory data; a second template tracking trajectory is obtained based on the tracking trajectory template dictionary and the first template tracking trajectory, wherein the second template tracking trajectory is a target template tracking trajectory, and the target template tracking trajectory consists of multiple target character sequences, and the target character sequences are character sequences extracted from the tracking trajectory template dictionary.

[0044] The second index, denoted as index D (template indecies), represents the set of indexes, specifically the set of span indexes. Spans are represented very similarly to logs, but are further constrained by the operations performed within the trace, meaning that changing parameters can cause noise in the representation. Similar to logs, applying template extraction techniques generates a representative set of template spans.

[0045] Index k is similar to t in the log template representation, representing the sequence number.

[0046] Ti represents the i-th observed trajectory. A single element in Ti, such as the basic span element, is added... <start>and <end>After constructing a richer span representation, we obtain a richer span template, where k represents the element index in the D (templateindecies) index set. Therefore, Ti represents a process that corresponds one-to-one with the index.

[0047] Specifically, distributed tracing is a request-centric approach used to describe behavior in distributed systems. The corresponding record data is called spans, which describe information about the operations performed by related services when handling external requests, including start time, end time, service name, and HTTP path. It can be represented as follows:

[0048]

[0049] in, This represents a specific tracking trajectory within a set of tracking data. represent The length of the trajectory or the number of spans in the trajectory. To provide a richer trajectory representation, the trajectory can be further enhanced. More specifically, two artificial spans ( <start>and <end>Accordingly, spans are added to the start and end points of the track. The spans represented in this form have a very similar representation to logs, with the additional constraint that spans are further limited by the operations performed within the track. This means they also face the problem of noise in the representation caused by varying parameters. Similar to logs, applying template extraction techniques can generate a set of representative template spans, allowing each track to be represented as a series of template spans.

[0050]

[0051] in, Based on the index Mapped to The extraction template, i.e. It can be observed that each function call is a character sequence, constructing a dictionary of character sequences appearing in a given set of trajectories. It provides a description of all spans appearing in the observed trajectory, formalized as:

[0052]

[0053] in It is a character sequence extracted from the span dictionary. Because spans have varying numbers of characters, an additional character was added to each span for consistent processing. <specspan>The index k is similar to t in the log template representation, representing the sequence number. Ti represents the i-th observation trajectory. A single element in Ti, such as representing the basic span element, is added... <start>and <end>After constructing a richer span representation, we obtain a richer span template, where k represents the element index in the D (template indeterminates) index set. Therefore, Ti represents a process that corresponds one-to-one with the index.

[0054] Step S206: Process the target template log and target template tracking trajectory through the target LSTM network to obtain target prediction log records and target prediction tracking trajectory data;

[0055] In the technical solution provided in step S206, the step of simultaneously processing the target template log and the target template tracking trajectory through the target LSTM network to obtain the predicted log record and the predicted tracking trajectory data includes: constructing a target prediction task model based on the target template log and the target template tracking trajectory, wherein the target prediction task model is used to determine the predicted log record and the predicted tracking trajectory data; constructing a target LSTM network based on the target prediction task model; and simultaneously processing the target template log and the target template tracking trajectory through the target LSTM network to obtain the predicted log record and the predicted tracking trajectory data.

[0056] Specifically, since the occurrence of the next log message depends on the occurrence of the previous log history, and in the tracking trajectory, the occurrence of the next span depends on the previous span, the problem of constructing a target prediction task model can be formulated as follows:

[0057]

[0058] in, Indicates from arrive Logs or tracking templates between them This indicates the size of the saved historical data. In this invention, the task model is called Next Template Prediction (NTP), where T represents time. win This indicates the latest time scale with the length of the win time.

[0059] In some embodiments of this application, the target LSTM network includes a template embedding layer, a first LSTM cluster, a second LSTM cluster, and a probability output layer. The template embedding layer is used to convert target template logs into target template log sequences and target template tracking trajectories into target template tracking trajectory sequences. The first LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template log sequences. The second LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template tracking trajectory sequences. The outputs of the first and second LSTM clusters are concatenated and fed back to the first and second LSTM clusters through an additional linear layer. The probability output layer is used to output prediction results, which include possible predicted log records and their probabilities of occurrence, as well as possible predicted tracking trajectory data and their probabilities of occurrence.

[0060] As an optional implementation, the probability output layer includes a first linear layer and a second linear layer, wherein the first linear layer is used to determine the prediction log record and the second linear layer is used to determine the prediction tracking trajectory data.

[0061] It should be noted that log data and tracking trajectory data, after being normalized as described above, have similar forms and are therefore processed using the same architecture. Firstly, multimodality is reflected in integrating information from both log data and tracking trajectory data to achieve anomaly detection. Secondly, as... Figure 3 The bidirectional LSTM unit shown here, compared to the traditional RNN which can only utilize past context information, uses two separate hidden layers to process past and future data and then forwards it to the same output layer to achieve a more accurate anomaly detection process.

[0062] Specifically, when the target template log and target template tracking trajectory are input into the Mobibao LSTM network, the following will be provided: and A dictionary composed of words is initialized using random vectors for each word of a specific size, with parameters used. Describes the initial size of the words. The template embedding layer uses word representations to create corresponding template sequences, which are input to the deep learning model LSTM. The model models the sequence correlations between input samples. A word representation refers to the input word vector initialized with random vectors, and the template sequence is a sequential description of the input word vectors. Finally, the template prediction result is output through a softmax layer. The specific calculation method is as follows:

[0063]

[0064] Here, the distribution of all possible templates is calculated. Given a sequence of template inputs, the template with the highest probability is considered the most likely template to appear. To fully utilize log data and tracking trajectory data to describe the state of this distributed system and to perform anomaly detection through an end-to-end system, this invention proposes a joint anomaly detection architecture based on multimodal LSTM. Specifically, at the input, the two types of data are merged, i.e., log and span dictionaries are provided simultaneously for each submodule in the model. Then, the outputs of the two LSTM clusters are concatenated and merged, and fed through an additional linear layer. Specifically, feeding through an additional linear layer means that the previously obtained features are connected to the subsequent decision layer through the corresponding linear activation function y=x. At the same time, each LSTM cluster consists of multiple bidirectional LSTMs, which allows the entire network to provide the advantage of including information from both modes and to make fuller use of the correlation between data when performing anomaly detection, thereby improving prediction performance. At the output layer, the shared information from the connections is passed through two linear layers, one for tracking trajectory data prediction and the other for log data prediction.

[0065] In some embodiments of this application, the step of constructing the target LSTM network further includes: constructing a target cost function and training the target LSTM network using the target cost function, wherein the output of the target cost function is the joint cross-entropy loss of the target prediction log record and the target prediction tracking trajectory data.

[0066] Specifically, in order to comprehensively consider the information contained in these two modes, the cost function can be designed to calculate the joint cross-entropy loss of the most likely log template and span template when given the joint input information over a certain period of time. The specific formula is as follows:

[0067]

[0068] in, Represents the classification cross-entropy loss. and This represents the true value of the next relevant log or span template. When the backpropagation step is complete, the loss function incorporates information from both modalities, so gradients are calculated based on this information to better predict anomalous states.

[0069] Step S208: Determine the operating status of the distributed system to be detected, and determine whether the distributed system to be detected is in an abnormal working state based on the target prediction log record, the target prediction tracking trajectory data and the operating status.

[0070] This application provides a method such as Figure 4 The diagram shows the architecture of the system anomaly detection process. Figure 4 As can be seen from the embodiments of this application, the system anomaly detection method can simultaneously process log data and tracking data, and output log prediction and tracking prediction, thereby obtaining prediction output. Specifically, when processing log data and tracking data, the log data and tracking data are first represented as a combination of words through word embedding, and then the log data and word data are converted into template sequences through template embedding and input into the corresponding LSTM clusters. The two LSTM clusters jointly process the log data and tracking data, and output log prediction results and tracking prediction results.

[0071] In some embodiments of this application, during execution Figure 2 The system anomaly detection method shown or Figure 4 The actual process of the system anomaly detection procedure shown is as follows: Figure 5 As shown, it includes the following steps:

[0072] Step S502: Input the log data and tracking trajectory data generated in the distributed system;

[0073] Step S504: Standardize the input log and tracking trajectory data.

[0074] Step S506: Construct a pseudo-task for anomaly detection;

[0075] Step S508: Construct an end-to-end deep learning architecture for solving NTP tasks: a bidirectional multimodal LSTM network.

[0076] Step S510: Output the corresponding logs and prediction results of the tracking trajectory, and determine whether an abnormal state has occurred based on the operating status of the distributed system.

[0077] This method involves acquiring log records and tracking trajectory data of the distributed system under test; standardizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, where the data formats of the target template logs and target template tracking trajectories are identical; processing the target template logs and target template tracking trajectories using a target LSTM network to obtain target predicted log records and target predicted tracking trajectory data; determining the operating status of the distributed system under test; and determining whether the distributed system under test is in an abnormal operating state based on the target predicted log records, target predicted tracking trajectory data, and operating status. By using a target LSTM network to process the target template logs and target template tracking trajectories to obtain target predicted log records and target predicted tracking trajectory data, this method achieves the goal of comprehensively considering log records and tracking trajectory data to determine whether the system is abnormal, thus achieving a high-accuracy technical effect for system anomaly detection. This solves the technical problem of low detection accuracy caused by using single data for system anomaly detection in related technologies.

[0078] This application provides a system anomaly detection device. Figure 6 This is a schematic diagram of the anomaly detection device of the system. For example... Figure 6 As shown, the device includes: an input module 60 for acquiring log records and tracking trajectory data of the distributed system to be detected; a standardization module 62 for standardizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and target template tracking trajectories are the same; a first processing module 64 for processing the target template logs and target template tracking trajectories through a target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; and a second processing module 66 for determining the operating status of the distributed system to be detected, and determining whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, the target prediction tracking trajectory data, and the operating status.

[0079] In some embodiments of this application, the log records include log constants and log variables. The steps of the standardization module 62 in standardizing the log records include: extracting log constants from the log records; establishing a log template dictionary based on the log constants; and obtaining a target template log based on the log template dictionary. The target template log consists of multiple target extraction words, which are extraction words determined from the log template dictionary based on the first index corresponding to the log record.

[0080] In some embodiments of this application, the tracking trajectory data consists of multiple span data, wherein the span data is used to record operation information when the distributed system to be detected processes external requests, and the operation information includes at least one of the following: start time, end time, service name, and HTTP path; the standardization module 62 performs the standardization processing of the tracking trajectory data by adding a first preset span at the starting point of the tracking trajectory data and adding a second preset span at the ending point of the tracking trajectory data, wherein the first preset span is used to indicate the starting point of the tracking trajectory data and the second preset span is used to indicate the ending point of the tracking trajectory data; after adding the first preset span and the second preset span, a first template tracking trajectory is obtained by using a second index and a trajectory template set, wherein the first template tracking trajectory includes multiple template spans determined from the trajectory template set according to the second index, and the second index is an index determined based on the tracking trajectory data; a second template tracking trajectory is obtained based on the tracking trajectory template dictionary and the first template tracking trajectory, wherein the second template tracking trajectory is a target template tracking trajectory, and the target template tracking trajectory consists of multiple target character sequences, and the target character sequences are character sequences extracted from the tracking trajectory template dictionary.

[0081] In some embodiments of this application, the first processing module 64 processes the target template log and the target template tracking trajectory through a target LSTM network to obtain predicted log records and predicted tracking trajectory data. The steps include: constructing a target prediction task model based on the target template log and the target template tracking trajectory, wherein the target prediction task model is used to determine the predicted log records and predicted tracking trajectory data; constructing a target LSTM network based on the target prediction task model; and simultaneously processing the target template log and the target template tracking trajectory through the target LSTM network to obtain the predicted log records and predicted tracking trajectory data.

[0082] In some embodiments of this application, the target LSTM network includes a template embedding layer, a first LSTM cluster, a second LSTM cluster, and a probability output layer. The template embedding layer is used to convert target template logs into target template log sequences and target template tracking trajectories into target template tracking trajectory sequences. The first LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template log sequences. The second LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template tracking trajectory sequences. The outputs of the first and second LSTM clusters are concatenated and fed back to the first and second LSTM clusters through an additional linear layer. The probability output layer is used to output prediction results, which include possible predicted log records and their probabilities of occurrence, as well as possible predicted tracking trajectory data and their probabilities of occurrence.

[0083] In some embodiments of this application, the probability output layer includes a first linear layer and a second linear layer, wherein the first linear layer is used to determine the predicted log record and the second linear layer is used to determine the predicted tracking trajectory data.

[0084] In some embodiments of this application, the first processing module 64 constructs a target LSTM network by: constructing a target cost function and training the target LSTM network using the target cost function, wherein the output of the target cost function is the joint cross-entropy loss of the target prediction log record and the target prediction tracking trajectory data.

[0085] This application provides a non-volatile storage medium storing a program. During program execution, the device containing the non-volatile storage medium performs the following system anomaly detection method: acquiring log records and tracking trajectory data of the distributed system to be detected; normalizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the target template logs and target template tracking trajectories have the same data format; processing the target template logs and target template tracking trajectories through a target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; determining the operating state of the distributed system to be detected, and determining whether the distributed system to be detected is in an abnormal operating state based on the target prediction log records, target prediction tracking trajectory data, and operating state.

[0086] This application provides an electronic device, including a memory and a processor. The processor runs a program stored in the memory, wherein the program executes the following system anomaly detection method: acquiring log records and tracking trajectory data of a distributed system to be detected; normalizing the log records and tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the target template logs and target template tracking trajectories have the same data format; processing the target template logs and target template tracking trajectories through a target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; determining the operating state of the distributed system to be detected, and determining whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, target prediction tracking trajectory data, and operating state.

[0087] It should be noted that each module in the above-mentioned system anomaly detection device can be a program module (for example, a set of program instructions that implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0088] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0089] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0094] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / end> < / start> < / specspan> < / end> < / start> < / end> < / start> < / speclog>

Claims

1. A method for detecting system anomalies, characterized in that, include: Obtain the log records and tracking trajectory data of the distributed system under test. The log records include log constants and log variables. The tracking trajectory data consists of multiple span data. The span data is used to record the operation information of the distributed system under test when processing external requests. The operation information includes at least one of the following: start time, end time, service name, and HTTP path. The log records and the tracking trajectory data are normalized to obtain the target template log and the target template tracking trajectory, wherein the data format of the target template log and the target template tracking trajectory is the same; The target template log and target template tracking trajectory are processed by the target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; Determine the operating status of the distributed system to be detected, and determine whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, the target prediction tracking trajectory data and the operating status; The steps for normalizing the log records include: Extract the log constants from the log records; A log template dictionary is created based on the aforementioned log constants; Based on the log template dictionary, the target template log is obtained, wherein the target template log is composed of multiple target extraction words, and the target extraction words are extraction words determined from the log template dictionary based on the first index corresponding to the log record; The step of normalizing the tracking trajectory data includes: A first preset span is added to the starting point of the tracking trajectory data, and a second preset span is added to the ending point of the tracking trajectory data, wherein the first preset span is used to indicate the starting point of the tracking trajectory data, and the second preset span is used to indicate the ending point of the tracking trajectory data; After adding the first preset span and the second preset span, a first template tracking trajectory is obtained by using a second index and a trajectory template set. The first template tracking trajectory includes multiple template spans determined from the trajectory template set based on the second index, where the second index is an index determined based on the tracking trajectory data. Based on the tracking trajectory template dictionary and the first template tracking trajectory, a second template tracking trajectory is obtained, wherein the second template tracking trajectory is the target template tracking trajectory, and the target template tracking trajectory is composed of multiple target character sequences, which are character sequences extracted from the tracking trajectory template dictionary.

2. The system anomaly detection method according to claim 1, characterized in that, The step of processing the target template log and target template tracking trajectory through the target LSTM network to obtain target prediction log records and target prediction tracking trajectory data includes: Based on the target template log and the target template tracking trajectory, a target prediction task model is constructed, wherein the target prediction task model is used to determine the target prediction log records and the target prediction tracking trajectory data; Based on the target prediction task model, construct the target LSTM network; The target template log and the target template tracking trajectory are processed simultaneously by the target LSTM network to obtain the target prediction log record and the target prediction tracking trajectory data.

3. The system anomaly detection method according to claim 2, characterized in that, The target LSTM network includes a template embedding layer, a first LSTM cluster, a second LSTM cluster, and a probability output layer, wherein... The template embedding layer is used to convert the target template log into a target template log sequence and to convert the target template tracking trajectory into a target template tracking trajectory sequence. The first LSTM cluster consists of multiple bidirectional LSTM units and is used to process the target template log sequence; The second LSTM cluster, composed of multiple bidirectional LSTM units, is used to process the target template tracking trajectory sequence. The outputs of the first LSTM cluster and the second LSTM cluster are concatenated and combined, and then fed back to the first LSTM cluster and the second LSTM cluster through an additional linear layer. The probability output layer is used to output prediction results, wherein the prediction results include possible target prediction log records and the probability of occurrence of the target prediction log records, as well as possible target prediction tracking trajectory data and the probability of occurrence of the target prediction tracking trajectory data.

4. The system anomaly detection method according to claim 3, characterized in that, The probability output layer includes a first linear layer and a second linear layer, wherein the first linear layer is used to determine the target prediction log record, and the second linear layer is used to determine the target prediction tracking trajectory data.

5. The system anomaly detection method according to claim 2, characterized in that, The steps for constructing the target LSTM network include: Construct a target cost function and train the target LSTM network using the target cost function, wherein the output of the target cost function is the joint cross-entropy loss of the target prediction log record and the target prediction tracking trajectory data.

6. A system anomaly detection device, characterized in that, include: The input module is used to acquire the log records and tracking trajectory data of the distributed system under test. The log records include log constants and log variables. The tracking trajectory data consists of multiple span data. The span data is used to record the operation information of the distributed system under test when processing external requests. The operation information includes at least one of the following: start time, end time, service name, and HTTP path. A standardization module is used to standardize the log records and the tracking trajectory data to obtain target template logs and target template tracking trajectories, wherein the data formats of the target template logs and the target template tracking trajectories are the same; The first processing module is used to process the target template log and the target template tracking trajectory through the target LSTM network to obtain target prediction log records and target prediction tracking trajectory data; The second processing module is used to determine the operating status of the distributed system to be detected, and to determine whether the distributed system to be detected is in an abnormal working state based on the target prediction log records, the target prediction tracking trajectory data and the operating status. The standardization module is further configured to extract the log constants from the log records; establish a log template dictionary based on the log constants; and obtain the target template log based on the log template dictionary, wherein the target template log consists of multiple target extraction words, and the target extraction words are extraction words determined from the log template dictionary based on the first index corresponding to the log record; The standardization module is further configured to add a first preset span at the starting point of the tracking trajectory data and a second preset span at the ending point of the tracking trajectory data, wherein the first preset span is used to indicate the starting point of the tracking trajectory data and the second preset span is used to indicate the ending point of the tracking trajectory data; after adding the first preset span and the second preset span, a first template tracking trajectory is obtained using a second index and a trajectory template set, wherein the first template tracking trajectory includes multiple template spans determined from the trajectory template set based on the second index, and the second index is an index determined based on the tracking trajectory data; a second template tracking trajectory is obtained based on the tracking trajectory template dictionary and the first template tracking trajectory, wherein the second template tracking trajectory is the target template tracking trajectory, and the target template tracking trajectory consists of multiple target character sequences, the target character sequences being character sequences extracted from the tracking trajectory template dictionary.

7. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the system anomaly detection method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the system anomaly detection method according to any one of claims 1 to 5.

Images