Abnormal log determination method and device, equipment, storage medium and program product

Abstract

The invention discloses an abnormal log determination method and device, equipment, a storage medium and a program product. The method comprises the steps that multiple first logs generated in the system running process are obtained, each first log comprises a host name, a process name, a timestamp and original log content, and the original log content comprises dynamic parameters; mapping dynamic parameters in the original log content into corresponding semantic categories to obtain target log content; determining a first log vector based on the host name, the process name, the timestamp and the target log content; sorting the plurality of first log vectors based on the timestamps to obtain a log sequence; determining an abnormal log vector in a plurality of first log vectors based on a normalized log sequence corresponding to the system and a context dependency relationship of the plurality of first log vectors in the log sequence; and determining the first log corresponding to the abnormal log vector as an abnormal log. According to the embodiment of the invention, the efficiency and accuracy of determining the abnormal log can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of operation and maintenance, and particularly relates to a method and device for determining abnormal logs, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Based on system running logs, fault diagnosis and positioning are performed, and accurate alarms are output accordingly, which is a key link for ensuring the stability of modern information systems, especially large distributed systems and micro-service architectures. Timely and accurate alarms can provide vital fault clues for operation and maintenance personnel, and are a key prerequisite for realizing rapid response and recovery and avoiding business losses.

[0003] However, with the popularity of micro-service architectures, the services, components and their dependencies within the system become increasingly complex. This complexity in the architecture directly leads to an exponential increase in system log data, forming a massive, high-speed and multi-source log stream. The traditional method of fault diagnosis and positioning based on system running logs has been difficult to adapt to the massive, high-speed and multi-source log stream, resulting in low efficiency and accuracy of fault diagnosis and positioning. SUMMARY

[0004] Embodiments of the application provide a method and device for determining abnormal logs, an electronic device, a computer readable storage medium and a computer program product, which can improve the efficiency and accuracy of determining abnormal logs, and further improve the efficiency and accuracy of fault diagnosis and positioning.

[0005] In a first aspect, embodiments of the application provide a method for determining abnormal logs, which comprises: obtaining a plurality of first logs generated in a system running process, wherein the first logs comprise a host name, a process name, a timestamp and original log content, and the original log content comprises dynamic parameters; mapping the dynamic parameters in the original log content to corresponding semantic categories to obtain target log content; determining a first log vector based on the host name, the process name, the timestamp and the target log content; sorting a plurality of first log vectors based on the timestamp to obtain a log sequence; determining an abnormal log vector in the plurality of first log vectors based on a normalized log sequence corresponding to the system and a context dependency relationship of the plurality of first log vectors in the log sequence; determining a first log corresponding to the abnormal log vector as an abnormal log.

[0006] In a possible implementation, the determining the first log vector based on the host name, the process name, the timestamp, and the target log content comprises: establishing a combination relationship between the host name and the process name to obtain a host-process pair; determining the first log vector based on the host-process pair, the timestamp, and the target log content.

[0007] In a possible implementation, the output frequency of the target log content corresponding to different systems is different. The method comprises: determining the number of acquired first logs based on the output frequency of the target log content; acquiring a plurality of first logs generated in the system running process based on the number of acquired first logs.

[0008] In a possible implementation, before the determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of first log vectors in the log sequence, the method further comprises: determining an adjacent log time interval based on the timestamps of two adjacent first log vectors in the log sequence; counting the occurrence frequency of the target log content in the log sequence; updating the first log vector based on the adjacent log time interval and the occurrence frequency of the target log content in the log sequence.

[0009] In a possible implementation, the updating the first log vector based on the adjacent log time interval and the occurrence frequency of the target log content in the log sequence comprises: performing normalization processing on a plurality of adjacent log time intervals in the log sequence to obtain a plurality of target time intervals; mapping the plurality of target time intervals into period encodings respectively; updating the first log vector based on the period encodings and the occurrence frequency of the target log content in the log sequence.

[0010] In a possible implementation, the determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of first log vectors in the log sequence comprises: dividing the log sequence into a plurality of sub-log sequences; For each of the sub-log sequences, a log vector prediction model is used to predict target log content in a next log vector after the sub-log sequence, to obtain a first probability corresponding to each of a plurality of preset log contents, the plurality of preset log contents including the target log content, the plurality of preset log contents being all target log contents included in the normalized log sequence, the log vector prediction model being trained based on a plurality of system respectively corresponding normalized log sequences; In a case where the first probability of the target log content is less than a probability threshold, a first log vector corresponding to the target log content is determined as an abnormal log vector.

[0011] In a possible implementation, the log vector prediction model includes a bidirectional long short-term memory network layer, an attention layer and a fully connected layer; the using the log vector prediction model to predict the target log content in the next log vector after the sub-log sequence to obtain the first probability corresponding to each of the plurality of preset log contents includes: The bidirectional long short-term memory network layer is used to extract time sequence features of the sub-log sequence, to obtain first time sequence features; The attention layer is used to perform dimension reduction processing on the first time sequence features, to obtain second time sequence features; The fully connected layer is used to map the second time sequence features to the first probability corresponding to each of the plurality of preset log contents.

[0012] In a possible implementation, before the using the log vector prediction model to predict the target log content in the next log vector after the sub-log sequence, the method further includes: A plurality of system respectively corresponding normalized log sequences are obtained; Each of the normalized log sequences is divided into a plurality of sub-normalized log sequences; For each of the sub-normalized log sequences, the sub-normalized log sequence is determined as a log sequence sample, and a next target log content after the sub-normalized log sequence is determined as a log content label corresponding to the log sequence sample; An initial log vector prediction model is used to predict target log content in a next log vector after the log sequence sample, to obtain a second probability corresponding to each of the plurality of preset log contents; Based on the second probability corresponding to each of the plurality of preset log contents and the log content label, a loss function value is determined; In a case where the loss function value is greater than a preset threshold, a model parameter of the initial log vector prediction model is adjusted, and a prediction of a target log content in a next log vector after the log sequence sample is performed by using the initial log vector prediction model until the loss function value is less than or equal to the preset threshold, to obtain the log vector prediction model.

[0013] In a possible implementation, different system versions correspond to different normalized log sequences; after the log vector prediction model is obtained, the method further includes: obtaining a plurality of log sequence samples and respective prediction result sets corresponding to the plurality of log sequence samples output by the log vector prediction model, the prediction result set including a plurality of preset log contents and respective prediction probabilities corresponding to the plurality of preset log contents; In a case where a system version is updated, determining a change of the normalized log sequence; based on the change of the normalized log sequence, adjusting at least one of a prediction probability and a preset log content in the prediction result set; In a case where the first probability of the target log content is less than a probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector. In a case where the first probability of the target log content is less than a probability threshold, and a prediction probability of the target log content in the prediction result set is less than the probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector.

[0014] In a possible implementation, after the dynamic parameter in the original log content is mapped to a corresponding semantic category to obtain a target log content, the method further includes: storing a correspondence between the first log and the dynamic parameter in the first log; After the first log corresponding to the abnormal log vector is determined as an abnormal log, the method further includes: based on the correspondence between the first log and the dynamic parameter, determining an abnormal dynamic parameter corresponding to the abnormal log; based on the abnormal dynamic parameter, restoring the target log content corresponding to the abnormal log to an original log content; analyzing the original log content corresponding to the abnormal log to determine an abnormal subject; querying a configuration item corresponding to the abnormal subject in a configuration management database; based on the configuration item corresponding to the abnormal subject, outputting first alarm information.

[0015] In a possible implementation, the configuration item corresponding to the abnormal subject includes an abnormal node, and the outputting of the first alarm information based on the configuration item corresponding to the abnormal subject includes: obtaining a plurality of candidate abnormal nodes having the same configuration information as the abnormal node; querying, in the configuration management database, configuration items corresponding to the plurality of candidate abnormal nodes respectively; outputting a plurality of first alarm information based on the configuration item corresponding to the abnormal subject and the configuration items corresponding to the plurality of candidate abnormal nodes respectively.

[0016] In a possible implementation, after the outputting of the plurality of first alarm information, the method further includes: obtaining second alarm information based on the first alarm information; determining an alarm type based on the second alarm information by using an alarm classification model; determining an alarm priority based on the second alarm information and the alarm type by using a bidirectional long short-term neural network model; processing alarm events corresponding to the plurality of second alarm information respectively based on the alarm priority.

[0017] In a possible implementation, the processing of the alarm event corresponding to the second alarm information includes: determining similarities between the alarm event and a plurality of historical alarm events respectively corresponding to historical alarm information of the plurality of historical alarm events based on the second alarm information and the historical alarm information; determining a target similarity that meets a similarity condition from the plurality of similarities; determining a target historical alarm event corresponding to the target similarity as the target historical alarm event; processing the alarm event corresponding to the second alarm information based on an alarm processing mode corresponding to the target historical alarm event.

[0018] In a second aspect, an embodiment of the present application provides a determination apparatus of an abnormal log, which comprises: an obtaining module, configured to obtain a plurality of first logs generated in a system running process, wherein the first log comprises a host name, a process name, a timestamp, and original log content, and the original log content comprises a dynamic parameter; a mapping module, configured to map the dynamic parameter in the original log content into a corresponding semantic category to obtain target log content; a determination module, configured to determine a first log vector based on the host name, the process name, the timestamp, and the target log content. The sorting module is configured to sort the first log vectors based on the time stamps to obtain a log sequence. The determining module is further configured to determine an abnormal log vector from the first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the first log vectors in the log sequence. The determining module is further configured to determine the first log corresponding to the abnormal log vector as an abnormal log.

[0019] In a third aspect, an electronic device is provided. The electronic device includes a processor and a memory storing computer program instructions. The processor, when executing the computer program instructions, implements the method in any possible implementation method of the first aspect.

[0020] In a fourth aspect, a computer readable storage medium is provided. The computer readable storage medium stores computer program instructions. When the computer program instructions are executed by a processor, the method in any possible implementation method of the first aspect is implemented.

[0021] In a fifth aspect, a computer program product is provided. Instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device executes the method in any possible implementation method of the first aspect.

[0022] The embodiment of the application can map the dynamic parameters in the original log content to corresponding semantic categories to obtain target log content, and determine a first log vector based on a host-process pair, a timestamp, and the target log content, so as to convert multiple first logs into multiple first log vectors, reduce data redundancy in the multiple first log vectors, and focus on the type of the log event itself when subsequent fault diagnosis and positioning are performed based on the first log vectors, without being disturbed by specific parameter values. In this way, the multiple first log vectors are sorted based on the timestamp to obtain a log sequence, and an abnormal first log vector in the multiple first log vectors is determined based on a context dependency relationship of the multiple first log vectors in the log sequence and with reference to a normalized log sequence corresponding to the system, so as to accurately identify a difference point (i.e., the abnormal first log vector) in the log sequence, and determine the first log corresponding to the abnormal first log vector as an abnormal log, thereby improving the accuracy of determining the abnormal log. In addition, the abnormal first log vector in the multiple first log vectors is determined based on the normalized log sequence corresponding to the system and the context dependency relationship of the multiple first log vectors in the log sequence, and the first log corresponding to the abnormal first log vector is determined as the abnormal log, instead of checking whether each first log is faulty, thereby improving the efficiency of determining the abnormal log. In this way, the efficiency and accuracy of determining the abnormal log can be improved by the embodiment of the application, and the efficiency and accuracy of fault diagnosis and positioning can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments of the application will be briefly introduced. Those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0024] Figure 1 FIG. 1 is a flow diagram of a method for determining an abnormal log according to an embodiment of the application; Figure 2 FIG. 2 is a structural diagram of an abnormal log determination apparatus according to an embodiment of the application; Figure 3 FIG. 3 is a structural diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0025] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. For the purpose of clarity, the description is divided into the following sections: technical solution, technical effects, and specific embodiments. The specific embodiments described are only intended for the purpose of providing a better understanding of the present application, and are not intended to limit the present application. The present application can be implemented without some of the specific details described below. The following description of the embodiments is merely provided to provide a better understanding of the present application by showing examples of the present application.

[0026] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... " does not, without more restrictions, preclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0027] It should be noted that in the embodiments of the present application, some software, components, models and the like in the industry may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the technical solution of the present application, but it does not mean that the applicant has or must have used the scheme.

[0028] As described in the background section, based on system running logs, fault diagnosis and positioning are carried out, and accurate alarms are output accordingly, which is a key link to ensure the stability of modern information systems, especially large distributed systems and micro-service architecture. Timely and accurate alarms can provide vital clues for fault diagnosis for operation and maintenance personnel, which is a key prerequisite for rapid response and recovery and avoiding business loss.

[0029] However, with the popularity of micro-service architecture, the services, components and their dependencies within the system become increasingly complex. This complexity in architecture directly leads to an exponential increase in system log data, forming a massive, high-speed, multi-source log stream. The traditional method of fault diagnosis and positioning based on system running logs has been difficult to adapt to the massive, high-speed, multi-source log stream, resulting in low efficiency and accuracy of fault diagnosis and positioning.

[0030] Therefore, to solve the problems in the related art, the embodiments of the present application provide a method and device for determining abnormal logs, an electronic device, a computer readable storage medium and a computer program product. The method for determining abnormal logs can be applied to an alarm platform. The alarm platform can include a fault detection device and an alarm processing device. The fault detection device can be configured to perform fault diagnosis and positioning based on massive, high-speed and multi-source log streams, and output first alarm information corresponding to the fault. The alarm processing device can be configured to process an alarm event based on the first alarm information.

[0031] In addition, the fault detection device can include a collection module, a semantic analysis module, an abnormality detection module and a fault positioning module. The collection module supports multi-source log access in structured, semi-structured and unstructured text formats, and can capture full-amount business logs such as micro-service calls and database operations in real time. The semantic analysis module is configured to decode log texts in depth, and extract log key information including timestamps, hostnames, process names and original log contents. The abnormality detection module introduces a normalized log sequence corresponding to the system as prior knowledge, and constructs a log vector detection model based on a bidirectional long short-term memory network, to capture abnormal sequence features and mine context semantic associations, and to identify implicit fault patterns. The fault positioning module is configured to locate modules and devices corresponding to abnormal logs based on configuration items of a configuration management database (CMDB), and output the first alarm information.

[0032] The method for determining abnormal logs provided by the embodiments of the present application will be described below.

[0033] Figure 1 A flowchart of the method for determining abnormal logs provided by an embodiment of the present application is shown. As shown in Figure 1 The method for determining abnormal logs provided by the embodiments of the present application includes the following steps: S110, a plurality of first logs generated in a system running process are acquired, the first logs including hostnames, process names, timestamps and original log contents, and the original log contents including dynamic parameters; S120, the dynamic parameters in the original log contents are mapped to corresponding semantic categories to obtain target log contents; S130, first log vectors are determined based on the hostnames, the process names, the timestamps and the target log contents; S140, the plurality of first log vectors are sorted based on the timestamps to obtain log sequences; S150, abnormal log vectors in the plurality of first log vectors are determined based on normalized log sequences corresponding to the system and context dependency relationships of the plurality of first log vectors in the log sequences; S160, determine the first log corresponding to the abnormal log vector as an abnormal log.

[0034] By mapping the dynamic parameters in the original log content to the corresponding semantic categories to obtain target log content, and determining the first log vector based on the host-process pair, the timestamp, and the target log content, the plurality of first logs can be converted into a plurality of first log vectors, the data redundancy in the plurality of first log vectors is reduced, and subsequent fault diagnosis and positioning based on the first log vector can focus on the type of log event itself without being disturbed by specific parameter values. In this way, by sorting the plurality of first log vectors based on the timestamp to obtain a log sequence, and taking the normalized log sequence corresponding to the system as a reference, the abnormal log vector in the plurality of first log vectors is determined based on the context dependency relationship of the plurality of first log vectors in the log sequence, the difference point (i.e. the abnormal log vector) in the log sequence can be accurately identified, and by determining the first log corresponding to the abnormal log vector as an abnormal log, the accuracy of determining the abnormal log can be improved. In addition, by determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of first log vectors in the log sequence, and determining the first log corresponding to the abnormal log vector as an abnormal log, rather than checking whether each first log is faulty, the efficiency of determining the abnormal log can be improved. In this way, by the embodiments of the present application, the efficiency and accuracy of determining the abnormal log can be improved, and the efficiency and accuracy of fault diagnosis and positioning can be improved.

[0035] The specific implementation of each of the above steps will be described below.

[0036] In some embodiments, in S110, the first log can be a log obtained by semi-structured conversion of the original log. The first log can be in the form of <host> - <process> - <timestamp> ： <msg>of the format presentation. Among them, <host>represents a host name, <process>represents a process name, <timestamp>representing a timestamp, <msg>The original log content is represented. The original log content can include dynamic parameters. The dynamic parameters refer to variable parts in the log template whose specific values vary with the context, state, user input or external events during program running. If the original log content is "Connection from 192.168.1.1 failed", "192.168.1.1" is a dynamic parameter. In addition, the target template is a static and common part extracted from a series of logs with different content but the same meaning. It represents a general pattern or structure of a class of log events, while ignoring the specific and variable parameters therein.

[0037] As an example, the embodiments of the present application can first continuously collect multi-source and heterogeneous original logs through a collection module, and then parse the original logs through a semantic analysis module to extract host name, process name, timestamp, original log content and the like fields and field values corresponding to the fields from the original logs, and then extract the target template according to the field values corresponding to the fields. <host> - <process> - <timestamp> ： <msg>The original log is converted into a semi-structured first log in the format of

[0038] Since different systems, middleware and applications (such as Hadoop clusters, Redis, Kubernetes components, etc.) not only have different log formats, but also have different storage paths and output methods, in order to collect the above-mentioned multi-source and heterogeneous original logs, various types of proprietary adapters can be integrated to parse the log format of specific components (such as Hadoop, Redis, K8s), and complete data grabbing from the specified file path or output stream. In order to ensure long-term and comprehensive log collection, the adapter supports automated deployment. In a containerized environment, the Kubernetes DaemonSet method is preferred for deployment, so that an adapter instance runs on each node in the cluster, and all original logs of the components on the node are collected without omission. This deployment mode ensures that the entire collection module can seamlessly cover the mixed infrastructure from the local data center to various cloud environments. Through a flexible and automatically deployable adapter, the complexity of the underlying log source is shielded, and a unified and stable log data stream is provided for the subsequent processing link.

[0039] In addition, the semantic analysis module can support the parsing of standard log files such as Request for Comments (RFC) 5424 standard, enterprise internal log specification, and user-defined log format. After collecting the original log, the semantic analysis module can be used to analyze the original log to extract the host name, process name, timestamp, original log content and other fields from the original log, as well as the field values corresponding to the fields.

[0040] After the host name, process name, timestamp, original log content and other fields and their field contents are parsed, the original log can be converted into a structured second log in the format of <host> - <process> - <timestamp> ： <msg>The original log is converted into a semi-structured first log in the format of

[0041] In addition, the embodiment of the present application can acquire a plurality of first logs according to a target window. The target window specifies the number of first logs to be acquired. The size of the target window can be denoted as w. If w = 100, 100 first logs can be acquired each time. If w = 50, 50 first logs can be acquired each time.

[0042] In some embodiments, in S120, different dynamic parameters can correspond to different semantic categories. For example, the dynamic parameter "192.168.1.1" can correspond to the semantic category "ip", and the dynamic parameter "3306" can correspond to the semantic category "port". By mapping the dynamic parameters in the original log content to their corresponding semantic categories, the original log content can be abstracted into target log content. The target log content can also be referred to as a log template. The plurality of target log contents may, for example, include "Connection from {ip} failed", "Looking", "My selection bind port: {ip} / {ip}:{port}", and the like.

[0043] In this way, in the implementation process, 20,000 original log contents can be abstracted into 51 target log contents.

[0044] In addition, it should be noted that for the same system, the target log content output by the system is usually a fixed number and constantly repeated in a certain order. For example, for a certain system, the target log content output by the system has three E1, E2, and E3 in total, and the three target log contents are constantly repeated in the order of E1, E3, E2, thereby forming a time sequence as follows: {E1, E3, E2, E1, E3, E2, E1, E3, E2……}. In the above time sequence, by adding the corresponding host name, process name, and timestamp information to each target log content, a normalized log sequence V1, V2, V3, V4, V5, V6, V7, V8, V9…… corresponding to the system can be obtained. That is, by the above method, the disordered multiple logs output by the system are processed into a normalized log sequence having a context-dependent relationship. The normalized log sequence can be used to provide a reference for determining abnormal logs.

[0045] In addition, the output frequency of the target log content corresponding to different systems can be different. That is, in the 100 logs of the A system obtained, some target log content can appear 25 times. In the 100 logs of the B system obtained, some target log content can appear 2 times. Among them, the target log content appearing 25 times can mean that the A system corresponds to 4 different target log contents in total, and the 4 target log contents are cycled 25 times in total. The target log content appearing 2 times can mean that the B system corresponds to 50 different target log contents in total, and the 50 target log contents are cycled 2 times in total.

[0046] Based on this, in order to improve the efficiency and accuracy of determining the abnormal log, in some embodiments, S110 can specifically include: determining the number of acquired first logs based on the output frequency of the target log content; acquiring a plurality of first logs generated in the system running process based on the number of acquired first logs.

[0047] Here, as described above, the embodiments of the present application can acquire a plurality of first logs according to the target window. Among them, the size of the target window can be automatically adjusted according to the output frequency of the target log content. For example, for a high-frequency system (such as the A system), the target window can be small, such as w=16, to avoid reducing the efficiency of determining the abnormal log due to the inclusion of too much redundant information in the plurality of first logs. In addition, for a low-frequency system (such as the B system), the target window can be large, such as w=200, to capture complete log context information and avoid reducing the accuracy of determining the abnormal log due to information fragmentation.

[0048] Therefore, by determining the number of acquired first logs based on the output frequency of the target log content, and acquiring a plurality of first logs generated in the system running process based on the number of acquired first logs, the rationality of acquiring the number of first logs can be improved, both avoiding information redundancy and avoiding information fragmentation, thereby improving the efficiency and accuracy of determining the abnormal log.

[0049] In addition, in order to restore the target log content corresponding to the abnormal log to the original log content after determining the abnormal log, and perform fault positioning based on the original log content corresponding to the abnormal log, in some embodiments, after obtaining the target log content, the correspondence between the first log and the dynamic parameter in the first log can also be stored. Therefore, after finally identifying the abnormal log, the dynamic parameter in the target log content corresponding to the abnormal log can be determined based on the correspondence, the original log content corresponding to the abnormal log is obtained, and fault positioning is performed based on the original log content corresponding to the abnormal log.

[0050] In some embodiments, in S130, after the original log content is converted into the target log content, for each first log, log information such as the host name, the process name, the timestamp, and the target log content can be extracted from the first log, and the log information is re-encoded to form a standardized dictionary, and a first log vector corresponding to the first log is obtained. For example, if the first log includes the host name "SERVER01", the process name "nginx", the timestamp [14, 30, 0], and the target log content "Connection from {ip} failed", the first log vector can be recorded as {"host": "SERVER01", "process": "nginx", "timestamp": [14, 30, 0], "msg": "Connection from {ip} failed"}.

[0051] In addition, there is usually a dependency relationship between the host and the process. In some actual cases, even if the host and the process are normal respectively, the combination of the host and the process can be abnormal. Therefore, in order to avoid the misjudgment caused by some processes only executing on the corresponding host node and improve the accuracy of determining the abnormal log, in some embodiments, S110 can specifically include: establishing a combination relationship between the host name and the process name to obtain a host-process pair; determining the first log vector based on the host-process pair, the timestamp, and the target log content.

[0052] If the first log includes the host name "SERVER01", the process name "nginx", the timestamp [14, 30, 0], and the target log content "Connection from {ip} failed", the first log vector can be recorded as {"host_process": "SERVER01_nginx", "timestamp": [14, 30, 0], "msg": "Connection from {ip} failed"}.

[0053] The embodiments of the present application can capture the dependency relationship between fields by establishing a combination between the host name and the process name in the form of splicing field values (such as "host_process" = "SERVER01_nginx"), avoid the misjudgment caused by some processes only executing on the corresponding host node, and thus improve the accuracy of determining the abnormal log.

[0054] In some embodiments, after determining the plurality of first log vectors in S140, the discrete log vectors can be converted into continuous time series by sorting the plurality of first log vectors based on the timestamps, obtaining a log sequence. The log sequence can be denoted as S t =[V t-w ,V t-w+1 ,…,V t ]。

[0055] In some embodiments, each system can correspond to a normalized log sequence in S150. As described above, the normalized log sequence can be denoted as {V1, V2, V3, V4, V5, V6, V7, V8, V9……}, for example. Wherein, V j may represent a single first log vector, and V j may include information such as host-process pairs, timestamps, and target log contents. In addition, the target log contents corresponding to each system can be fixed in number and constantly repeated in a certain order. For example, if the target log contents corresponding to the system have E1, E2, E3 in total, and the three target log contents constantly repeat in the order of E1, E3, E2, then only from the target log contents, a target log content sequence corresponding to the normalized log sequence {E1, E3, E2, E1, E3, E2, E1, E3, E2……} can be obtained. That is, there is a context-dependent relationship between the plurality of log vectors in the normalized log sequence in the arrangement order of the target log contents.

[0056] On this basis, after obtaining the log sequence, by analyzing the context-dependent relationship of the plurality of first log vectors in the log sequence, especially the time series relationship of the target log contents in the plurality of first log vectors, the abnormal points in the log sequence relative to the normalized log sequence can be determined, and then the first log vector corresponding to the abnormal point is determined as the abnormal log vector.

[0057] Based on this, in order to improve the accuracy of determining the abnormal log vector, in some embodiments, the above S150 can specifically include: dividing the log sequence into a plurality of sub-log sequences; for each sub-log sequence, using a log vector prediction model to predict the target log content in the next log vector after the sub-log sequence, obtaining a first probability corresponding to each of a plurality of preset log contents, the plurality of preset log contents including the target log content, the plurality of preset log contents being all target log contents included in the normalized log sequence, the log vector prediction model being trained based on the normalized log sequences corresponding to the plurality of systems respectively; in a case where the first probability of the target log content is less than a probability threshold, determining the first log vector corresponding to the target log content as the abnormal log vector.

[0058] Here, the log vector prediction model can be constructed based on a bidirectional long short-term memory network (Bi-LSTM). The log vector prediction model can be trained based on the normalized log sequences respectively corresponding to the plurality of systems. Therefore, the log vector prediction model has learned the correct context dependency relationship respectively corresponding to the plurality of normalized log sequences. In addition, the plurality of preset log contents can be all target log contents respectively corresponding to the plurality of systems outputting the plurality of first logs.

[0059] As an example, the plurality of sub-log sequences can be obtained by dividing the log sequence based on a sliding window. The sliding window can be a fixed-size window. By moving the sliding window, the plurality of first log vectors in the log sequence can be traversed, avoiding omissions and ensuring the comprehensiveness of the abnormal log vector detection. Specifically, the embodiments of the present application can move the sliding window according to a preset step size to divide the log sequence into a plurality of sub-log sequences. The preset step size can be 1, for example. For example, assuming that the log sequence is {V’1, V’2, V’3, V’4, V’5, V’6, V’7, V’8, V’9, …}, the size of the sliding window is 3, and the preset step size is 1, the plurality of sub-log sequences can include {V’1, V’2, V’3}, {V’2, V’3, V’4}, {V’3, V’4, V’5}, {V’4, V’5, V’6}, and so on.

[0060] Since the log vector prediction model has learned the correct context dependency relationship respectively corresponding to the plurality of normalized log sequences, for each sub-log sequence, the log vector prediction model can use the learned knowledge to predict the target log content in the next log vector after the sub-log sequence, obtain a first probability respectively corresponding to each of the plurality of preset log contents, and find a first probability corresponding to the target log content among the first probabilities respectively corresponding to the plurality of preset log contents. If the first probability of the target log content is greater than or equal to a probability threshold, it can be determined that the context dependency relationship between the first log vector corresponding to the target log content and the sub-log sequence is the same as the context dependency relationship corresponding to the normalized log sequence learned by the log vector prediction model, and thus it can be determined that the first log vector corresponding to the target log content is a normal log vector. Conversely, if the first probability of the target log content is less than the probability threshold, it can be determined that the context dependency relationship between the first log vector corresponding to the target log content and the sub-log sequence is different from the context dependency relationship corresponding to the normalized log sequence learned by the log vector prediction model, and thus it can be determined that the first log vector corresponding to the target log content is an abnormal log vector.

[0061] As an example, the log vector prediction model can include a bidirectional long short-term memory network layer and a fully connected layer. Based on this, the above prediction of the target log content in the next log vector after the sub-log sequence by the log vector prediction model to obtain the first probability corresponding to each of the plurality of preset log contents can specifically include: extracting the time sequence features of the sub-log sequence by the bidirectional long short-term memory network layer to obtain first time sequence features; mapping the first time sequence features to the first probability corresponding to each of the plurality of preset log contents by the fully connected layer.

[0062] The embodiments of the present application can improve the accuracy of determining the abnormal log vector by pre-training the log vector prediction model based on the normalized log sequences corresponding to the plurality of systems respectively, and detecting the abnormal log vector based on the log vector prediction model.

[0063] Based on this, in order to further improve the accuracy of determining the abnormal log vector, in some embodiments, the log vector prediction model can also include an attention layer. Based on this, the above prediction of the target log content in the next log vector after the sub-log sequence by the log vector prediction model to obtain the first probability corresponding to each of the plurality of preset log contents can specifically include: extracting the time sequence features of the sub-log sequence by the bidirectional long short-term memory network layer to obtain first time sequence features; performing dimensionality reduction processing on the first time sequence features by the attention layer to obtain second time sequence features; mapping the second time sequence features to the first probability corresponding to each of the plurality of preset log contents by the fully connected layer.

[0064] Here, the attention layer can first calculate an importance score for each time step in the first time sequence features; then normalize the importance score to obtain a normalized attention weight; and then weight sum the first time sequence features by the attention weight to obtain the second time sequence features. Compared with the first time sequence features, the second time sequence features have lower dimension and contain more important information.

[0065] In this way, by adding an attention layer after the bidirectional long short-term memory network layer, performing dimensionality reduction processing on the first time sequence features by the attention layer to obtain the second time sequence features, and mapping the second time sequence features to the first probability corresponding to each of the plurality of preset log contents by the fully connected layer, the accuracy of determining the abnormal log vector can be further improved.

[0066] In addition, in order to further improve the accuracy of determining the abnormal log vector, in some embodiments, after the above S140 and before S150, the method for determining the abnormal log can further include: determine an adjacent log time interval based on timestamps of two adjacent first log vectors in the log sequence; count a frequency of occurrence of the target log content in the log sequence; update the first log vector based on the adjacent log time interval and the frequency of occurrence of the target log content in the log sequence.

[0067] Here, for two adjacent first log vectors in the log sequence, if a timestamp of a previous first log vector is t i-1 , and a timestamp of a next first log vector is t i , the adjacent log time interval corresponding to the next first log vector can be T i =t i -t i-1 .

[0068] In addition, the frequency of occurrence of the target log content in the log sequence can be denoted as F i . For example, for the first log vector V i , if the frequency of occurrence of the target log content corresponding to the first log vector in the log sequence is 3, it can be determined that the value of F i is 3.

[0069] In this way, by splicing the adjacent log time interval, the frequency of occurrence of the target log content in the log sequence, and the first log vector before updating, the updated first log vector can be obtained. For example, if the first log vector before updating is denoted as V i , based on the adjacent log time interval T i and the frequency of occurrence of the target log content in the log sequence F i , the updated first log vector obtained by updating the V i may be denoted as [V i , T i , F i ].

[0070] In addition, if the adjacent log time interval and the frequency of occurrence of the target log content in the log sequence are included in the first log vector, correspondingly, the adjacent log time interval and the frequency of occurrence of the target log content in the normalized log sequence are also included in the log vector of the normalized log sequence.

[0071] The embodiments of the present application can strengthen the continuity of the log sequence and improve the importance of the target log content by updating the first log vector based on the adjacent log time interval and the frequency of occurrence of the target log content in the log sequence. In this way, by determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of updated first log vectors in the log sequence, the accuracy of determining the abnormal log vector can be further improved.

[0072] On this basis, in order to further improve the accuracy of determining the abnormal log vector, in some embodiments, the above updating the first log vector based on the adjacent log time interval and the occurrence frequency of the target log content in the log sequence can specifically include: normalizing the plurality of adjacent log time intervals in the log sequence to obtain a plurality of target time intervals; mapping the plurality of target time intervals into period encodings respectively; updating the first log vector based on the period encodings and the occurrence frequency of the target log content in the log sequence.

[0073] Here, the period encoding can better express the periodicity or time pattern feature of the first log vector relative to the adjacent log time interval, thereby solving the interference of the common "burst" phenomenon in the log data on the model's understanding of the time law.

[0074] In this way, by first normalizing the adjacent log time interval to obtain the target time interval, then mapping the target time interval into the period encoding, and updating the first log vector based on the period encoding and the occurrence frequency of the target log content in the log sequence, the periodic distortion problem that burst-intensive logs often occur can be solved, and the interference of the common "burst" phenomenon in the log data on the model's understanding of the time law can be avoided. In this way, by determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of updated first log vectors in the log sequence, the accuracy of determining the abnormal log vector can be further improved.

[0075] In addition, in order to obtain the log vector prediction model, in some embodiments, before the above prediction of the target log content in the next log vector after the sub-log sequence by the log vector prediction model, the abnormal log determination method can further include a model training process. The model training process can include the following steps: obtaining a plurality of normalized log sequences respectively corresponding to a plurality of systems; dividing each normalized log sequence into a plurality of sub-normalized log sequences; for each sub-normalized log sequence, determining the sub-normalized log sequence as a log sequence sample, and determining the next target log content after the sub-normalized log sequence as a log content label corresponding to the log sequence sample; using the initial log vector prediction model to predict the target log content in the next log vector after the log sequence sample, to obtain a second probability corresponding to each of a plurality of preset log contents; determining a loss function value based on the second probability corresponding to each of the plurality of preset log contents and the log content label; In a case where the loss function value is greater than the preset threshold, the model parameters of the initial log vector prediction model are adjusted, and the target log content in the next log vector after the log sequence sample is predicted by using the initial log vector prediction model is executed until the loss function value is less than or equal to the preset threshold, and the log vector prediction model is obtained.

[0076] The embodiments of the present application can move the sliding window according to a preset step length to divide the normalized log sequence into a plurality of sub-normalized log sequences. The size of the sliding window can be the same as the size of the sliding window when the log sequence is divided into sub-log sequences. The preset step length can be 1, for example.

[0077] In addition, based on the entire probability distribution (including the second probability corresponding to each of the plurality of preset log contents) output by the initial log vector prediction model and the one-hot encoding of the log content label, the cross-entropy loss between the probability distribution and the log content label can be calculated to obtain the loss function value. The one-hot encoding of the log content label indicates which is the only correct next target log content.

[0078] In this way, by adjusting the model parameters of the initial log vector prediction model in a case where the loss function value is greater than the preset threshold, and iteratively executing the above steps of predicting the second probability and determining the loss function value, until the loss function value is less than or equal to the preset threshold, a trained log vector prediction model can be obtained.

[0079] In addition, as the system is continuously updated, a plurality of system versions can be obtained. The system update can include online hosts and offline hosts. Therefore, the hosts corresponding to different system versions can be different. With the adjustment of the host, the host-process pair will be adjusted, and the normalized log sequence will also be adjusted accordingly. Therefore, different system versions can correspond to different normalized log sequences.

[0080] In a case where the system version is updated, the above trained log vector prediction model can be fine-tuned based on the latest normalized log sequence corresponding to the system version to ensure the accuracy of predicting abnormal log vectors using the log vector prediction model. However, fine-tuning the log vector prediction model requires a certain amount of time. In the case of high-speed output of the log stream, the time for fine-tuning the log vector prediction model may affect the accuracy of the model in predicting abnormal log vectors.

[0081] Based on this, in order to improve the efficiency of determining abnormal log vectors while ensuring the accuracy of determining abnormal log vectors, in some embodiments, after the above log vector prediction model is obtained, the method for determining abnormal logs can further include: obtaining a plurality of log sequence samples and their respective corresponding prediction result sets output by the log vector prediction model, the prediction result set comprising a plurality of preset log contents and their respective corresponding prediction probabilities; In the case of system version update, the change of the normalized log sequence is determined; Based on the change of the normalized log sequence, at least one of the prediction probability and the preset log content in the prediction result set is adjusted.

[0082] Based on this, in the case that the first probability of the target log content is less than the probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector, comprising: In the case that the first probability of the target log content is less than the probability threshold, and the prediction probability of the target log content in the prediction result set is less than the probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector.

[0083] Here, in the case that the loss function value is less than or equal to the preset threshold, the initial log vector prediction model can be determined as the log vector prediction model, and at the same time, a plurality of log sequence samples and their respective corresponding prediction result sets are output, each prediction result set comprising a plurality of preset log contents and their respective corresponding prediction probabilities. The prediction probability is the second probability when the initial log vector prediction model is determined as the log vector prediction model.

[0084] On this premise, if the system version is updated, the latest normalized log sequence can be obtained first, and then the latest normalized log sequence is compared with the previous normalized log sequence to determine the change of the normalized log sequence. Based on the change of the normalized log sequence, at least one of the prediction probability and the preset log content in the prediction result set is adjusted.

[0085] For example, if the probability distribution of the prediction result set corresponding to the log sequence sample A output by the model is as shown in Table 1: Table 1

[0086] Based on the change of the normalized log sequence, it is determined that the target log content in the next log vector of the log sequence sample A should be E1, and the probability of E1 in Table 1 can be adjusted to 1 to obtain Table 2: Table 2

[0087] If based on the change of the normalized log sequence, it is determined that the target log content in the next log vector of the log sequence sample A should be E4, E4 with a probability of 1 can be added in Table 1 to obtain Table 3: Table 3

[0088] As an example, assuming that the probability threshold is 0.9, and based on the change of the normalized log sequence, the target log content in the next log vector of the log sequence sample A should be E1, and the target log content in the next log vector of the log sequence B identical to the log sequence sample A is exactly E1, then the next log vector of the log sequence B is a normal log vector.

[0089] Without updating the model, when predicting the target log content in the next log vector of the log sequence B using the log vector prediction model, the model may predict that the first probability of E1 is 0.13, the first probability of E2 is 0.9823, and the first probability of E3 is 0.964. Since the first probability of E1 is less than 0.9, the next log vector of the log sequence B is determined as an abnormal log vector only by model prediction.

[0090] In fact, after updating the prediction result set, the abnormal log vector is predicted based on the model and the prediction result set. Specifically, as shown in Table 2, the prediction probability of E1 is 1, which is greater than 0.9, so it can be determined that the first log vector corresponding to E1, i.e., the next log vector of the log sequence B, is a normal log vector.

[0091] In this way, by determining the first log vector corresponding to the target log content as an abnormal log vector when the first probability of the target log content is less than the probability threshold and the prediction probability of the target log content in the prediction result set is less than the probability threshold, the accuracy of determining the abnormal log vector can be ensured. Since the prediction result set is updated based on the change of the normalized log sequence, rather than retraining the model, the time for model training is saved, and the efficiency of determining the abnormal log vector is improved.

[0092] In addition, it should be noted that the above-mentioned determination of the abnormal log vector based on the updated prediction result set can be an emergency measure. In order to determine the long-term stability of the abnormal log vector, the log vector prediction model will still be fine-tuned based on the latest normalized log sequence at appropriate times.

[0093] In some embodiments, in S160, after determining the abnormal log vector, the first log corresponding to the abnormal log vector can be determined as an abnormal log.

[0094] On this basis, in order to handle the fault in time and ensure the normal operation of the system, in some embodiments, after the above S160, the method can further include; determining an abnormal dynamic parameter corresponding to the abnormal log based on the correspondence between the first log and the dynamic parameter in the first log. restore the target log content corresponding to the abnormal log to the original log content based on the abnormal dynamic parameter; analyze the original log content corresponding to the abnormal log to determine an abnormal subject; query a configuration item corresponding to the abnormal subject in the configuration management database; output first alarm information based on the configuration item corresponding to the abnormal subject.

[0095] Here, as described above, after mapping the dynamic parameter in the original log content to its corresponding semantic category to obtain the target log content, the correspondence between the first log and the dynamic parameter in the first log is also stored. Therefore, after determining the abnormal log, the abnormal dynamic parameter corresponding to the abnormal log can be determined based on the correspondence between the first log and the dynamic parameter in the first log, and the target log content corresponding to the abnormal log can be restored to the original log content based on the abnormal dynamic parameter. Then, analyzing the msg syntax structure of the original log content can identify the abnormal subject corresponding to the abnormal log. For example, performing core predicate and argument role identification on msg="Connection from 192.168.1.1 failed” can obtain the abnormal subject 192.168.1.1 (agent) and the action type Connection (network action).

[0096] On this basis, the interface of the configuration management database can be called to query the configuration item corresponding to the abnormal subject to obtain the application system, node, etc. to which the abnormal subject (such as 192.168.1.1) belongs, i.e., the host name, system, component, module, etc. of the fault location, and generate and output the first alarm information based on the above ip address, host name, system, component, module, etc.

[0097] The embodiments of the present application can automatically or remind the operation and maintenance personnel to handle the fault in time to ensure the normal operation of the system by restoring the target log content to the original log content based on the correspondence between the first log and the dynamic parameter in the first log, querying the configuration item corresponding to the abnormal subject in the configuration management database, and outputting the first alarm information based on the configuration item corresponding to the abnormal subject.

[0098] In addition, the node to which the abnormal subject belongs can be an abnormal node, i.e., the configuration item corresponding to the abnormal subject can include the abnormal node.

[0099] Therefore, in order to improve the comprehensiveness of the alarm and provide more comprehensive information for subsequent fault troubleshooting and processing, and improve the reliability and timeliness of fault processing, in some embodiments, the above outputting the first alarm information based on the configuration item corresponding to the abnormal subject can specifically include: obtain a plurality of candidate abnormal nodes having the same configuration information as the abnormal node; query configuration items corresponding to the plurality of candidate abnormal nodes respectively in the configuration management database; output a plurality of first alarm information based on the configuration item corresponding to the abnormal subject and the configuration items corresponding to the plurality of candidate abnormal nodes respectively.

[0100] The embodiments of the present application can improve the comprehensiveness of the alarm by outputting not only the first alarm information corresponding to the abnormal node but also the first alarm information corresponding to the plurality of candidate abnormal nodes having the same configuration information as the abnormal node after determining the abnormal node, thereby providing more comprehensive information for subsequent troubleshooting and processing, and improving the reliability and timeliness of fault processing.

[0101] In addition, the embodiments of the present application can also generate a topology structure diagram of the fault based on the information of the abnormal node and the plurality of candidate abnormal nodes, and recommend an emergency plan for preferentially using the abnormal node or a certain candidate abnormal node in combination with the topology structure diagram, so as to preferentially process the fault of the abnormal node or a certain candidate abnormal node and improve the fault processing efficiency.

[0102] In addition, it should be noted that, with the continuous refinement of system monitoring rules, not only the first alarm information corresponding to the abnormal node is output after determining the abnormal node, but also the first alarm information corresponding to the plurality of candidate abnormal nodes having the same configuration information as the abnormal node is output, so that the number of first alarm information increases explosively, and the number of generated event sheets also increases day by day. In the face of massive event sheets, it is difficult for operation and maintenance personnel to quickly and accurately classify alarm events, determine processing priorities, and select appropriate processing schemes.

[0103] Therefore, the embodiments of the present application can also perform intelligent classification, priority prediction, processing scheme recommendation and the like of alarm events by an alarm processing device according to the plurality of first alarm information, so as to improve the processing efficiency of alarm events and reduce operation and maintenance costs.

[0104] Therefore, the embodiments of the present application can also perform intelligent classification, priority prediction, processing scheme recommendation and the like of alarm events by an alarm processing device according to the plurality of first alarm information, so as to improve the processing efficiency of alarm events and reduce operation and maintenance costs. obtain second alarm information based on the first alarm information; determine an alarm type based on the second alarm information by an alarm classification model; determine an alarm priority based on the second alarm information and the alarm type by a bidirectional long short-term neural network model; process alarm events corresponding to the plurality of second alarm information respectively based on the alarm priority.

[0105] Here, the fault detection apparatus can also generate an event sheet corresponding to the alarm synchronously when outputting the first alarm information. The event sheet can be a structured table for recording information such as alarm occurrence time, alarm influence range, alarm cause, etc. The information in the event sheet can be derived from the first alarm information.

[0106] After receiving the first alarm information output by the fault detection apparatus, the alarm processing apparatus can also query other data sources based on the host name, alarm event, influence range (i.e. the above-mentioned multiple candidate abnormal nodes) in the first alarm information as conditions, thereby providing a basis for further alarm diagnosis and processing. Specifically, the alarm processing apparatus can query monitoring data based on the host name and alarm time as conditions. The monitoring data can include memory usage , central processing unit (CPU) usage , etc. In addition, the alarm processing apparatus can also query the information in the above-mentioned event sheet based on the host name or influence range (such as "10 host computers"). In addition, the alarm processing apparatus can also obtain a processing process log corresponding to the alarm event based on the host name as a condition. The processing process log can include processing time, processing personnel, operation instruction, processing step, etc.

[0107] The above-mentioned monitoring data, information in the event sheet, processing process log and first alarm information can be collectively referred to as second alarm information.

[0108] After obtaining the above-mentioned multiple sources of second alarm information, the second alarm information can be first preprocessed to improve the data quality of the second alarm information. The data preprocessing can include data cleaning, numerical normalization and text preprocessing.

[0109] Specifically, data cleaning can be to remove invalid data by using an outlier detection algorithm, for example, to identify CPU usage outliers by Specifically, outliers exceeding can be determined by the following formula (1) and removed: (1) In formula (1), μ is the CPU usage mean, is the standard deviation.

[0110] In addition, for numerical indicators such as , , the following formula (2) can be used to map the numerical normalization to the interval [0, 1]: (2) In formula (2), x is​​ , the original value of the numerical index, , the minimum value and the maximum value of the index, respectively.

[0111] The text preprocessing can be for the text of the alarm reason, after the text is segmented by using a segmentation tool, the text can be converted into a feature vector based on a Term Frequency-Inverse Document Frequency (TF-IDF) algorithm as shown in formula (3): (3) In formula (3), is the number of times that the word w appears in the document d, and N is the total number of documents, is the number of documents containing the word w, is the term frequency of the word w in the document d, is the inverse document frequency of the word w.

[0112] In addition, the alarm classification model can be a deep learning model constructed based on a Convolutional Neural Network (CNN). CNN is a deep learning model specially used for processing data with grid-like topology. The structure of the alarm classification model can be as follows: input layer (dimension , corresponding to the second alarm information after preprocessing) → 2 layers of convolutional layers (Conv2D, convolution kernel size , activation function ReLU) → 1 layer of pooling layer (MaxPooling2D, pooling kernel ) → 2 layers of fully connected layers (Dense) → output layer (Softmax, 12 output nodes corresponding to 12 alarm events). Among them, the output layer can calculate the probability P(k) of the alarm event belonging to the k-th class by the following formula (4), and the class corresponding to the maximum value of P(k) is the final classification result: (4) In formula (4), is the logit value of the k-th event output by the fully connected layer.

[0113] In this way, by inputting the second alarm information after preprocessing into the alarm classification model, the alarm type can be output.

[0114] In addition, when training the alarm classification model, in order to improve the training efficiency, the learning rate of the model can be set to 0.001, the batch size can be set to 32, the number of iterations (Epoch) can be set to 50, the optimizer can be determined as Adaptive Moment Estimation (Adam), and the loss function used to optimize the model parameters can be a cross-entropy loss function, as shown in the following formula (5): (5) In formula (5), is the true label (if the event belongs to the kth class , otherwise ), and m is the number of training samples.

[0115] In addition, the training data of the alarm classification model can include a plurality of preprocessed second alarm information samples and their respective corresponding alarm type labels.

[0116] In the embodiments of the present application, the alarm type can also be referred to as an alarm event type and an event type.

[0117] In addition, the alarm priority can be divided into three levels: P1 (urgent, red highlight, need to be handled within 30 minutes), P2 (important, orange highlight, need to be handled within 2 hours), and P3 (general, yellow highlight, need to be handled within 4 hours). The bidirectional long short-term neural network model can be used for alarm priority prediction.

[0118] The bidirectional long short-term neural network model is a special recurrent neural network specially designed to solve the gradient vanishing or explosion problem encountered by traditional recurrent neural networks when processing long sequence data. It can effectively learn and remember long-term dependencies. The model structure of the bidirectional long short-term neural network model can be as follows: input layer (dimension 32x8, corresponding to 8-dimensional time sequence features of alarm type, impact range, processing time, alarm IP, alarm hostname, alarm system, alarm component, alarm module, etc., time step 32) → 2-layer LSTM layer (hidden unit number 64, activation function tanh) → fully connected layer (output 3 nodes corresponding to 3 priority levels). Among them, the priority prediction logic is: in the 3 probability values P(P1), P(P2), P(P3) output by the model, the priority corresponding to the maximum value is taken as the prediction result. For example, if P(P1)=0.85, P(P2)=0.12, and P(P3)=0.03, the alarm priority of the alarm event is predicted to be P1 level, triggering a red highlight reminder.

[0119] Thus, by inputting the alarm type, and the impact range, processing time consumption, alarm IP, alarm host name, alarm belonging system, alarm belonging component, alarm belonging module, etc. in the second alarm information into the bidirectional long short-term neural network model, the alarm priority can be output.

[0120] In addition, when training the alarm classification model, in order to improve the training efficiency, the learning rate of the model can be set to 0.0005, the batch size can be set to 64, the number of iterations (Epoch) can be set to 80, and the optimizer can be determined as Adam with Weight decay (AdamW).

[0121] In addition, the training data of the alarm classification model can include a plurality of time sequence feature sequence samples and their respective priority labels. Each time sequence feature sequence sample can include 8-dimensional time sequence features such as alarm type, impact range, processing time consumption, alarm IP, alarm host name, alarm belonging system, alarm belonging component, and alarm belonging module.

[0122] After determining the alarm priority, the alarm events corresponding to the plurality of second alarm information can be processed based on the alarm priority, that is, the alarm events with higher alarm priority indicated by the alarm priority are preferentially processed.

[0123] The embodiments of the present application can accurately select different models for different task features, use CNN model for alarm classification, utilize the advantage of CNN in capturing local features, accurately identify key patterns in event description, and realize efficient and accurate automatic classification. In addition, the LSTM model is used for priority prediction, which can utilize the advantage of LSTM in processing sequence dependency, integrate context features such as event type, occurrence time, and impact range, and realize dynamic and objective priority evaluation, replacing subjective and artificial judgment. In this way, a full-automatic pipeline from alarm type identification to importance determination is realized, and the results are visualized by red, orange, and yellow three colors, directly guiding the processing queue.

[0124] Based on this, in order to improve the efficiency of alarm processing, in some embodiments, the processing of the alarm event corresponding to the second alarm information can specifically include: Based on the second alarm information and the historical alarm information corresponding to the plurality of historical alarm events respectively, determine the similarity between the alarm event and the plurality of historical alarm events respectively; Determine a target similarity that meets a similarity condition among the plurality of similarities; Determine the historical alarm event corresponding to the target similarity as a target historical alarm event; Process the alarm event corresponding to the second alarm information based on the alarm processing mode corresponding to the target historical alarm event.

[0125] Here, the historical alarm information and alarm processing mode corresponding to the plurality of historical alarm events respectively can be stored in the operation and maintenance knowledge base. The operation and maintenance knowledge base can also include device manuals, fault processing guidelines, expert experience documents and the like. In this way, when the above text is preprocessed, the documents in the knowledge base can also be text preprocessed to convert the text into a feature vector. The specific text preprocessing method can be referred to the related description above, and will not be described here.

[0126] When the similarity is calculated, the alarm type, alarm IP, alarm host name, alarm system, alarm component, alarm module and the like in the second alarm information and the historical alarm information are mainly used. The alarm type, alarm IP, alarm host name, alarm system, alarm component, alarm module and the like in the second alarm information can be processed into the first event feature sequence, and the alarm type, alarm IP, alarm host name, alarm system, alarm component, alarm module and the like in the historical alarm information can be processed into the second event feature sequence.

[0127] On this basis, the embodiment of the present application can determine the target historical alarm event using a Transformer-based recommendation model. The Transformer-based recommendation model is a deep learning architecture with a self-attention mechanism as the core. The Transformer-based recommendation model can have the following model structure: input layer (first event feature sequence, second event feature sequence, length 50, dimension 128) → encoder (6 layers, each layer containing multi-head attention layer, layer normalization, feedforward network) → decoder (6 layers, structure same as encoder) → output layer (similarity score calculation module).

[0128] When the specific similarity matching calculation is performed, the first event feature sequence corresponding to the new alarm event can be calculated by the following formula (6) and the second event feature sequence corresponding to the historical alarm event The cosine similarity between them is: (6) In formula (6), d is the feature vector dimension corresponding to the event feature sequence, , are the i-th dimensional elements of , respectively.

[0129] On this basis, the top 3 similarities with the maximum value can be determined as target similarities satisfying the similarity condition, the historical alarm event corresponding to the target similarity can be determined as a target historical alarm event, and the processing scheme corresponding to the target historical alarm event can be pushed to the operation and maintenance personnel or system as a recommendation result. The operation and maintenance personnel or the system automatically processes the alarm event corresponding to the second alarm information based on the alarm processing mode corresponding to the recommendation result. The output of the recommendation result can be as follows: for example, the new alarm event is "server CPU usage rate suddenly rises (=98%)", the model calculates that the similarity between it and the historical alarm event "20250610-001 (CPU occupation is too high, processing scheme: restart process A + optimize scheduling strategy)" is 0.92, and the scheme and the corresponding operation and maintenance manual fragment ("process A restart command: systemctl restart A") are pushed to the operation and maintenance personnel.

[0130] In addition, when training the recommendation model based on the Transformer, in order to improve the training efficiency, the learning rate of the model can be set to 0.0001, the batch size can be set to 128, the number of iterations (Epoch) can be set to 100, and the optimizer can be determined as Adam with Weight decay (AdamW).

[0131] The embodiments of the present application introduce the advanced model Transformer in the natural language processing field into the operation and maintenance scene. The core attention mechanism can deeply understand the semantic association between historical events and new events, realize "accurate similarity matching" instead of simple keyword retrieval, and thus can recommend the most relevant processing scheme, expert experience or manual fragment from a large number of historical cases and knowledge bases in seconds, greatly improving the fault troubleshooting efficiency and realizing the effective reuse of operation and maintenance knowledge.

[0132] In summary, the embodiments of the present application break through the limitation of a single data source and propose a preprocessing procedure for unified cleaning, tokenization and normalization of monitoring alarms (time series data), event records (label data), processing logs (process data) and knowledge base texts (unstructured data), forming a high-quality, multi-dimensional training data set, laying a solid data foundation for subsequent high-precision models, and realizing "explanation of governance" of operation and maintenance data.

[0133] In addition, through the automatic analysis and processing of operation and maintenance data by deep learning technology, intelligent classification of events is realized, time-consuming and errors of manual sorting are avoided; meanwhile, the priority of the event is accurately predicted, resources are preferentially dispatched to handle high-priority events, response delay is reduced, targeted processing scheme recommendation is matched, the fault troubleshooting and solution cycle is further shortened, and the overall event processing efficiency is greatly improved.

[0134] In addition, the embodiments of the present application do not need to rely on manual analysis, classification and scheme formulation of massive operation and maintenance data, thereby reducing the labor cost; and through priority prediction and scheme recommendation, the waste of resources caused by improper processing sequence and scheme trial and error is reduced, thereby indirectly reducing the time and resource cost loss in the operation and maintenance process.

[0135] In addition, based on intelligent analysis driven by deep learning, event classification, priority determination and scheme recommendation are all generated based on data rules, avoiding the subjectivity and limitations of manual experience judgment, thereby providing objective and reliable decision support for operation and maintenance personnel and improving the accuracy and standardization of operation and maintenance work.

[0136] With 5000+ daily monitoring alarms and 800+ event single records generated in a data center, covering 12 types of operation and maintenance events such as server hardware failure, network congestion, storage input / output (Input / Output, IO) timeout, application database exception, etc., after 1 month of trial operation of the scheme in the data center, the key indicators are as shown in Table 4: Table 4

[0137] The test results show that the scheme effectively solves the problems of manual dependence, strong subjectivity and low efficiency in traditional operation and maintenance, and significantly improves the intelligent level and decision accuracy of operation and maintenance event processing.

[0138] In addition, in order to avoid the decay of model performance over time, in some embodiments, new operation and maintenance data (about 35,000) will be collected every week, and the preprocessed data will be divided into training set, validation set and test set according to "7:2:1". When the performance of the model on the test set decreases by more than 5% (such as the event classification accuracy from 96% to 90%), fine-tuning is started, and incremental training method (only updating the last 2 layers of the model) is used to reduce the training time.

[0139] In this way, by periodically introducing new data to fine-tune the model, the entire system has the ability to continuously learn, can adapt to new operation and maintenance environments and event types, avoids the decay of model performance over time, and ensures the long-term effectiveness of the system.

[0140] Based on the above-mentioned embodiment of the method for determining abnormal logs, correspondingly, the present application also provides a specific implementation manner of a device for determining abnormal logs. Please refer to the following embodiments.

[0141] As shown in Figure 2 The device 200 for determining abnormal logs provided by an embodiment of the present application includes the following modules: The acquisition module 210 is configured to acquire a plurality of first logs generated in a system running process, the first log comprising a host name, a process name, a timestamp, and original log content, and the original log content comprising a dynamic parameter; The mapping module 220 is configured to map the dynamic parameter in the original log content into a corresponding semantic category to obtain target log content. The determination module 230 is configured to determine a first log vector based on the host name, the process name, the timestamp, and the target log content. The sorting module 240 is configured to sort a plurality of first log vectors based on the timestamp to obtain a log sequence. The determination module 230 is further configured to determine an abnormal log vector in the plurality of first log vectors based on a normalized log sequence corresponding to the system and a context dependency relationship of the plurality of first log vectors in the log sequence. The determination module 230 is further configured to determine a first log corresponding to the abnormal log vector as an abnormal log.

[0142] The determination device 200 for the abnormal log is described in detail as follows: In some embodiments, the determination module 230 can specifically include: The combination sub-module is configured to establish a combination relationship between the host name and the process name to obtain a host-process pair. The determination sub-module is configured to determine the first log vector based on the host-process pair, the timestamp, and the target log content.

[0143] In some embodiments, the output frequency of the target log content corresponding to different systems is different. Based on this, the acquisition module 210 can specifically include: The determination sub-module is further configured to determine the number of acquired first logs based on the output frequency of the target log content. The acquisition sub-module is configured to acquire a plurality of first logs generated in a system running process based on the number of acquired first logs.

[0144] In some embodiments, the determination device 200 for the abnormal log can further include: The determination module 230 is further configured to determine an adjacent log time interval based on the timestamps of two adjacent first log vectors in the log sequence before determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of first log vectors in the log sequence. The statistical module is configured to count the occurrence frequency of the target log content in the log sequence. The update module is configured to update the first log vector based on the adjacent log time interval and the occurrence frequency of the target log content in the log sequence.

[0145] In some embodiments, the updating module can specifically include: a normalization submodule configured to normalize a plurality of adjacent log time intervals in the log sequence to obtain a plurality of target time intervals; a mapping submodule configured to map the plurality of target time intervals into periodic encodings respectively; an updating submodule configured to update the first log vector based on the periodic encodings and the occurrence frequency of the target log content in the log sequence.

[0146] In some embodiments, the determining module 230 can specifically include: a division submodule configured to divide the log sequence into a plurality of sub-log sequences; a prediction submodule configured to, for each sub-log sequence, predict the target log content in the next log vector after the sub-log sequence by using a log vector prediction model to obtain a first probability corresponding to each of a plurality of preset log contents, the plurality of preset log contents including the target log content, the plurality of preset log contents being all target log contents included in the normalized log sequence, the log vector prediction model being trained based on a plurality of system respectively corresponding normalized log sequences; the determining submodule is further configured to determine the first log vector corresponding to the target log content as an abnormal log vector in a case where the first probability of the target log content is less than a probability threshold.

[0147] In some embodiments, the log vector prediction model includes a bidirectional long short-term memory network layer, an attention layer, and a fully connected layer. Based on this, the prediction submodule can specifically include: an extraction unit configured to extract time sequence features of the sub-log sequence by using the bidirectional long short-term memory network layer to obtain first time sequence features; a dimension reduction unit configured to reduce the dimension of the first time sequence features by using the attention layer to obtain second time sequence features; a mapping unit configured to map the second time sequence features into the first probability corresponding to each of the plurality of preset log contents by using the fully connected layer.

[0148] In some embodiments, the abnormal log determining apparatus 200 can further include: the acquisition module 210 is further configured to acquire a plurality of system respectively corresponding normalized log sequences before predicting the target log content in the next log vector after the sub-log sequence by using the log vector prediction model; a division module configured to divide each normalized log sequence into a plurality of sub-normalized log sequences; The determination module 230 is further configured to determine, for each sub-normalized log sequence, the sub-normalized log sequence as a log sequence sample, and a next target log content after the sub-normalized log sequence as a log content label corresponding to the log sequence sample. The prediction module is configured to predict, by using the initial log vector prediction model, the target log content in the next log vector after the log sequence sample, to obtain a second probability corresponding to each of a plurality of preset log contents. The determination module 230 is further configured to determine a loss function value based on the second probability corresponding to each of the plurality of preset log contents and the log content label. The iteration module is configured to, in a case where the loss function value is greater than a preset threshold, adjust a model parameter of the initial log vector prediction model, and return to perform the prediction of the target log content in the next log vector after the log sequence sample by using the initial log vector prediction model, until the loss function value is less than or equal to the preset threshold, to obtain a log vector prediction model.

[0149] In some embodiments, different system versions correspond to different normalized log sequences. Based on this, the abnormal log determination apparatus 200 can further include: The acquisition module 210 is further configured to, after obtaining the log vector prediction model, acquire a plurality of log sequence samples and a prediction result set corresponding to each of the plurality of log sequence samples output by the log vector prediction model, the prediction result set including a plurality of preset log contents and a prediction probability corresponding to each of the plurality of preset log contents. The determination module 230 is further configured to, in a case where a system version is updated, determine a change of the normalized log sequence. The adjustment module is configured to adjust at least one of the prediction probability and the preset log content in the prediction result set based on the change of the normalized log sequence. The determination module 230 is further configured to, in a case where the first probability of the target log content is less than a probability threshold, determine the first log vector corresponding to the target log content as an abnormal log vector, including: The determination module 230 is further configured to, in a case where the first probability of the target log content is less than a probability threshold, and a prediction probability of the target log content in the prediction result set is less than the probability threshold, determine the first log vector corresponding to the target log content as an abnormal log vector.

[0150] In some embodiments, the abnormal log determination apparatus 200 can further include: The storage module is configured to, after mapping the dynamic parameter in the original log content to the semantic category corresponding to the dynamic parameter to obtain the target log content, store a corresponding relationship between the first log and the dynamic parameter in the first log.

[0151] Based on this, the abnormal log determination apparatus 200 can further include: The determination module 230 is further configured to, after determining the first log corresponding to the abnormal log vector as the abnormal log, determine an abnormal dynamic parameter corresponding to the abnormal log based on a correspondence between the first log and the dynamic parameter; The restoration module is configured to restore target log content corresponding to the abnormal log to original log content based on the abnormal dynamic parameter; The analysis module is configured to analyze the original log content corresponding to the abnormal log to determine an abnormal subject; The query module is configured to query a configuration item corresponding to the abnormal subject in a configuration management database; The alarm module is configured to output first alarm information based on the configuration item corresponding to the abnormal subject.

[0152] In some embodiments, the configuration item corresponding to the abnormal subject includes an abnormal node. Based on this, the alarm module can specifically include: The acquisition sub-module is further configured to acquire a plurality of candidate abnormal nodes having the same configuration information as the abnormal node; The query sub-module is configured to query configuration items corresponding to the plurality of candidate abnormal nodes respectively in the configuration management database; The alarm sub-module is configured to output a plurality of first alarm information based on the configuration item corresponding to the abnormal subject and the configuration items corresponding to the plurality of candidate abnormal nodes respectively.

[0153] In some embodiments, the determination apparatus 200 of the abnormal log can further include: The acquisition module 210 is further configured to, after outputting the plurality of first alarm information, acquire second alarm information based on the first alarm information; The determination module 230 is further configured to determine an alarm type based on the second alarm information through an alarm classification model; The determination module 230 is further configured to determine an alarm priority based on the second alarm information and the alarm type through a bidirectional long short-term neural network model; The processing module is configured to process alarm events corresponding to the plurality of second alarm information respectively based on the alarm priority.

[0154] In some embodiments, the processing module can specifically include: The determination sub-module is further configured to determine a similarity between the alarm event and a plurality of historical alarm events respectively based on the second alarm information and historical alarm information corresponding to the plurality of historical alarm events respectively; The determination sub-module is further configured to determine a target similarity meeting a similarity condition from the plurality of similarities; The determination sub-module is further configured to determine a historical alarm event corresponding to the target similarity as a target historical alarm event; The processing submodule is configured to process an alarm event corresponding to the second alarm information based on an alarm processing mode corresponding to a target historical alarm event.

[0155] By mapping the dynamic parameters in the original log content to corresponding semantic categories to obtain target log content, and determining the first log vector based on the host-process pair, the timestamp, and the target log content, the plurality of first logs can be converted into a plurality of first log vectors, the data redundancy in the plurality of first log vectors is reduced, and subsequent fault diagnosis and positioning based on the first log vector can focus on the type of the log event itself, without being disturbed by specific parameter values. In this way, by sorting the plurality of first log vectors based on the timestamp to obtain a log sequence, and determining an abnormal log vector in the plurality of first log vectors based on the context dependency relationship of the plurality of first log vectors in the log sequence with reference to the normalized log sequence corresponding to the system, the difference point (i.e., the abnormal log vector) in the log sequence can be accurately identified, so that the abnormal log corresponding to the abnormal log vector can be determined as an abnormal log, thereby improving the accuracy of determining the abnormal log. In addition, by determining the abnormal log vector in the plurality of first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of the plurality of first log vectors in the log sequence, and determining the first log corresponding to the abnormal log vector as the abnormal log, rather than checking whether each first log is faulty, the efficiency of determining the abnormal log can be improved. In this way, by the embodiments of the present application, the efficiency and accuracy of determining the abnormal log can be improved, and the efficiency and accuracy of fault diagnosis and positioning can be improved.

[0156] Based on the method for determining an abnormal log provided in the above embodiments, the present application further provides a specific implementation of an electronic device. Figure 3 FIG. 1 shows a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0157] As shown in FIG. 1, the electronic device 300 can include a processor 310 and a memory 320 storing computer program instructions. Figure 3

[0158] Specifically, the processor 310 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits implementing the embodiments of the present application.

[0159] ​The memory 320 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 320 can include a Hard Disk Drive (HDD), a floppy disk drive, a flash memory, a Compact Disc (CD) or other optical disk, a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 320 can be removable and / or non-removable (or fixed) as appropriate. The memory 320 can be internal or external as appropriate. In particular embodiments, the memory 320 is non-volatile solid-state memory.

[0160] In particular embodiments, the memory 320 can be implemented using a Read Only Memory (ROM), a Random Access Memory (RAM), a static storage device, a dynamic storage device, etc. The memory 320 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the relevant program codes are saved in the memory 320 and are executed by the processor 310. The processor 310 reads and executes the computer program instructions stored in the memory 320 to implement any one of the abnormal log determination methods in the above embodiments.

[0161] The processor 310 reads and executes the computer program instructions stored in the memory 320 to implement any one of the abnormal log determination methods in the above embodiments.

[0162] In one example, the electronic device 300 can further include a communication interface 330 and a bus 340. As shown, the processor 310, the memory 320, the communication interface 330 are connected through the bus 340 and complete the communication between each other. Figure 3

[0163] The communication interface 330 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiments of the present application.

[0164] ​Bus 340 includes a hardware, software, or both that couples components of electronic device to each other. As an example and not by way of limitation, bus can include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a Hyper Transport (HT) interconnect, an industry standard architecture (ISA) bus, an InfiniBand (IB) interconnect, a low pin count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards board (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 340 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0165] By way of example, electronic device 300 can be a phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), among others.

[0166] The electronic device can execute the method for determining an exception log in the embodiments of this application, thereby achieving the advantages of the method for determining an exception log described above and having corresponding method embodiments, which will not be repeated here. Figure 1 The method for determining an exception log described above and having corresponding method embodiments, which will not be repeated here.

[0167] In addition, in combination with the method for determining the abnormal log in the above embodiments, the embodiments of the present application can provide a computer readable storage medium for implementation. The computer readable storage medium stores computer program instructions; the computer program instructions are executed by a processor to implement any of the methods for determining the abnormal log in the above embodiments. Examples of the computer readable storage medium include non-transitory computer readable storage media, such as a Read-Only Memory (ROM).

[0168] The computer program instructions stored in the storage medium of the above embodiments are used to enable the computer to perform the method for determining the abnormal log as shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0169] In combination with the method for determining the abnormal log in the above embodiments, the embodiments of the present application can provide a computer program product for implementation. The instructions in the computer program product are executed by the processor of an electronic device to implement any of the methods for determining the abnormal log in the above embodiments.

[0170] The computer program product of the above embodiments is used to implement the method for determining the abnormal log as shown in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0171] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0172] The functional blocks shown in the structural block diagrams described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine readable medium" can include any medium capable of storing or transmitting information. Examples of the machine readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, Radio Frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0173] It should also be noted that the example embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0174] The above describes the aspects of the present application with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each block in the flowcharts and / or block diagrams, and the combination of the blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagrams and / or flowcharts, and the combination of the blocks in the block diagrams and / or flowcharts, can also be implemented by special hardware that performs the specified functions or actions, or can be implemented by a combination of special hardware and computer instructions.

[0175] The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, modules and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. It should be understood that the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.< / msg> < / timestamp> < / process> < / host> < / msg> < / timestamp> < / process> < / host> < / msg> < / timestamp> < / process> < / host> < / msg> < / timestamp> < / process> < / host>

Claims

1. A method for determining anomaly logs, characterized in that, include: The system acquires multiple first log entries generated during system operation. Each first log entry includes a hostname, process name, timestamp, and original log content, which includes dynamic parameters. The dynamic parameters in the original log content are mapped to their corresponding semantic categories to obtain the target log content; A first log vector is determined based on the hostname, the process name, the timestamp, and the target log content; Based on the timestamp, the multiple first log vectors are sorted to obtain a log sequence; Based on the normalized log sequence corresponding to the system and the contextual dependencies of multiple first log vectors in the log sequence, abnormal log vectors among the multiple first log vectors are determined; The first log corresponding to the abnormal log vector is identified as the abnormal log.

2. The method according to claim 1, characterized in that, The step of determining the first log vector based on the hostname, the process name, the timestamp, and the target log content includes: Establish a combination relationship between the hostname and the process name to obtain a host-process pair; The first log vector is determined based on the host-process pair, the timestamp, and the target log content.

3. The method according to claim 1, characterized in that, The output frequency of target log content varies across different systems; the acquisition of multiple first log entries generated during system operation includes: The number of first log entries to be acquired is determined based on the output frequency of the target log content. Based on the aforementioned number of entries, multiple first log entries generated during system operation are obtained.

4. The method according to claim 1, characterized in that, Before determining the abnormal log vector among the multiple first log vectors based on the normalized log sequence corresponding to the system and the contextual dependencies of multiple first log vectors in the log sequence, the method further includes: The time interval between adjacent logs is determined based on the timestamps of two adjacent first log vectors in the log sequence; Count the frequency of occurrence of the target log content in the log sequence; The first log vector is updated based on the adjacent log time interval and the frequency of occurrence of the target log content in the log sequence.

5. The method according to claim 4, characterized in that, The step of updating the first log vector based on the adjacent log time interval and the frequency of occurrence of the target log content in the log sequence includes: The log time intervals of multiple adjacent logs in the log sequence are normalized to obtain multiple target time intervals; The multiple target time intervals are respectively mapped to periodic codes; The first log vector is updated based on the periodic encoding and the frequency of occurrence of the target log content in the log sequence.

6. The method according to claim 1, characterized in that, The step of determining the abnormal log vector among the multiple first log vectors based on the normalized log sequence corresponding to the system and the contextual dependencies of multiple first log vectors in the log sequence includes: The log sequence is divided into multiple sub-log sequences; For each sub-log sequence, the target log content in the next log vector following the sub-log sequence is predicted using a log vector prediction model to obtain the first probability corresponding to each of the multiple preset log contents. The multiple preset log contents include the target log content. The multiple preset log contents are all the target log contents included in the normalized log sequence. The log vector prediction model is trained based on the normalized log sequences corresponding to multiple systems. If the probability of the target log content is less than a probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector.

7. The method according to claim 6, characterized in that, The log vector prediction model includes a bidirectional long short-term memory network layer, an attention layer, and a fully connected layer. The log vector prediction model is used to predict the target log content in the next log vector following the sub-log sequence, obtaining a first probability corresponding to multiple preset log contents, including: The first temporal feature is obtained by extracting temporal features from the sub-log sequence using the bidirectional long short-term memory network layer. The attention layer is used to perform dimensionality reduction on the first temporal feature to obtain the second temporal feature; The second temporal feature is mapped to the first probability corresponding to each of the multiple preset log contents using a fully connected layer.

8. The method according to claim 6, characterized in that, Before using the log vector prediction model to predict the target log content in the next log vector following the sub-log sequence, the method further includes: Obtain the normalized log sequences corresponding to multiple systems; Each normalized log sequence is divided into multiple sub-normalized log sequences; For each of the sub-normalized log sequences, the sub-normalized log sequence is determined as a log sequence sample, and the next target log content after the sub-normalized log sequence is determined as the log content tag corresponding to the log sequence sample. The target log content in the next log vector after the log sequence sample is predicted using the initial log vector prediction model to obtain the second probability corresponding to each of the multiple preset log contents; Based on the second probability corresponding to each of the multiple preset log contents and the log content label, the loss function value is determined; If the loss function value is greater than a preset threshold, the model parameters of the initial log vector prediction model are adjusted, and the process of using the initial log vector prediction model to predict the target log content in the next log vector after the log sequence sample is repeated until the loss function value is less than or equal to the preset threshold, thus obtaining the log vector prediction model.

9. The method according to claim 8, characterized in that, Different system versions correspond to different normalized log sequences; after obtaining the log vector prediction model, the method further includes: Obtain multiple log sequence samples output by the log vector prediction model and their respective prediction result sets, wherein the prediction result sets include the multiple preset log contents and their respective prediction probabilities; In the event of a system version update, determine the changes in the normalized log sequence; Based on the changes in the normalized log sequence, adjust at least one of the prediction probability in the prediction result set and the preset log content; When the first probability of the target log content is less than a probability threshold, determining the first log vector corresponding to the target log content as an abnormal log vector includes: If the first probability of the target log content is less than a probability threshold, and the predicted probability of the target log content in the prediction result set is less than the probability threshold, the first log vector corresponding to the target log content is determined as an abnormal log vector.

10. The method according to any one of claims 1-9, characterized in that, After mapping the dynamic parameters in the original log content to their corresponding semantic categories to obtain the target log content, the method further includes: Store the correspondence between the first log and the dynamic parameters in the first log; After determining the first log corresponding to the abnormal log vector as an abnormal log, the method further includes: Based on the correspondence between the first log and the dynamic parameters, the abnormal dynamic parameters corresponding to the abnormal log are determined; Based on the aforementioned abnormal dynamic parameters, the target log content corresponding to the abnormal log is restored to the original log content; Parse the original log content corresponding to the anomaly log to determine the anomaly subject; Search the configuration item corresponding to the abnormal subject in the configuration management database; Based on the configuration items corresponding to the abnormal subject, the first alarm information is output.

11. The method according to claim 10, characterized in that, The configuration items corresponding to the abnormal subject include abnormal nodes. The step of outputting first alarm information based on the configuration items corresponding to the abnormal subject includes: Obtain multiple candidate abnormal nodes that have the same configuration information as the abnormal node; Query the configuration items corresponding to the multiple candidate abnormal nodes in the configuration management database; Based on the configuration items corresponding to the abnormal subject and the configuration items corresponding to the multiple candidate abnormal nodes, multiple first alarm messages are output.

12. The method according to claim 11, characterized in that, After outputting multiple first alarm messages, the method further includes: Based on the first alarm information, obtain the second alarm information; Based on the second alarm information, the alarm type is determined using an alarm classification model. Based on the second alarm information and the alarm type, the alarm priority is determined using a bidirectional long short-term neural network model. Based on the alarm priority, the alarm events corresponding to each of the multiple second alarm messages are processed.

13. The method according to claim 12, characterized in that, The alarm event corresponding to the second alarm message is processed, including: Based on the second alarm information and the historical alarm information corresponding to multiple historical alarm events, the similarity between the alarm event and the multiple historical alarm events is determined. Determine the target similarity that satisfies the similarity condition among the multiple similarities; The historical alarm events corresponding to the target similarity are identified as target historical alarm events; Based on the alarm processing method corresponding to the target historical alarm events, the alarm events corresponding to the second alarm information are processed.

14. A device for determining anomaly logs, characterized in that, The device includes: The acquisition module is used to acquire multiple first logs generated during system operation. The first logs include hostname, process name, timestamp, and original log content, and the original log content includes dynamic parameters. The mapping module is used to map the dynamic parameters in the original log content to their corresponding semantic categories to obtain the target log content. The determination module is used to determine a first log vector based on the hostname, the process name, the timestamp, and the target log content; The sorting module is used to sort multiple first log vectors based on the timestamps to obtain a log sequence; The determining module is further configured to determine the abnormal log vector among the multiple first log vectors based on the normalized log sequence corresponding to the system and the context dependency relationship of multiple first log vectors in the log sequence; The determining module is further configured to determine the first log corresponding to the abnormal log vector as an abnormal log.

15. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the method for determining the exception log as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the method for determining an exception log as described in any one of claims 1-13.

17. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the method for determining the exception log as described in any one of claims 1-13.

Images