Enhanced anomaly detection in computing environments

Abstract

Anomaly service receives log data from a node in a computing environment, which includes a sequence of information indicative of log messages generated by the node. The anomaly service identifies a dominant pattern of non-anomalous blocks representing the log messages in the sequence of information. Upon identifying the dominant pattern, the service is able to extract non-anomalous blocks from the log data to reveal anomalous blocks that do not conform to the dominant pattern. The service can then generate an anomaly vector based on the anomalous blocks, which can be distributed to the node to detect anomalies.

Description

Background Technology

[0001] Modern information services process large amounts of data, generating significant amounts of log data in the process. Log data is produced by runtime code to provide a record of the state of one or more components of the service. Log data can be useful in troubleshooting or otherwise maintaining the service. Examples of log data include log statements and operational metrics, both of which can be analyzed to predict abnormal conditions or events, such as processing failures, server failures, and computer hardware failures.

[0002] Anomalies can be detected and predicted by analyzing large amounts of log data showing patterns related to past anomalies in statements or metrics. Patterns can then be deployed against the data, triggering mitigation actions or alerts based on the presence of a given pattern. Unfortunately, this pattern extraction requires massive amounts of log data, which is very expensive to transport and process.

[0003] In fact, to successfully extract useful patterns, the amount of log data that needs to be sent from the edge to the central server can easily approach the amount of normal operational data sent in the same direction—assuming bandwidth exists to do so. Furthermore, the computational cost required to find patterns within a reasonable timeframe may exceed the computational cost initially allocated to normal operations. To date, these limitations have hindered the development and deployment of effective anomaly analysis. Summary of the Invention

[0004] This paper discloses techniques for improving anomaly detection and prediction in computing environments. In various implementations, an anomaly service receives log data from edge nodes in the computing environment, including information sequences indicating log messages generated by the nodes. The anomaly service identifies a dominant pattern in the information sequence representing non-anomaly blocks of log messages. After identifying the dominant pattern, the service can extract non-anomaly blocks from the log data to reveal anomaly blocks that do not conform to the dominant pattern. The service can then generate anomaly vectors based on the anomaly blocks, which can be distributed to edge nodes for anomaly detection.

[0005] In the same or other implementations, one or more nodes in the computing environment receive anomaly vectors. When an event occurs in the computing environment, a log message is generated, and in response, the nodes generate a corresponding sequence of hash values. A sequence vector is generated based on the hash value sequence, which can be compared with or otherwise evaluated against the anomaly vector to determine whether the log message indicates the occurrence of one or more anomalous events.

[0006] This overview is provided to introduce some concepts in a simplified form, which will be further described in the following technical disclosure. It is understood that this overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0007] Many aspects of this disclosure can be better understood with reference to the following accompanying drawings. While several embodiments are described in conjunction with these drawings, this disclosure is not limited to the embodiments disclosed herein. Rather, it is intended to cover all alternatives, modifications, and equivalents.

[0008] Figure 1 The diagram illustrates the operational system architecture in the implementation.

[0009] Figure 2 The diagram illustrates the log generation process in the implementation.

[0010] Figure 3 The diagram illustrates the exception block extraction process in the implementation.

[0011] Figure 4 The diagram illustrates the exception detection and handling process in the implementation.

[0012] Figure 5 The diagram illustrates the operational scenarios in the implementation.

[0013] Figure 6 The diagram illustrates a log generation scenario in the implementation.

[0014] Figure 7 The illustration shows an exception block extraction scenario in the implementation.

[0015] Figure 8 The abnormal block extraction scenario is further illustrated.

[0016] Figures 9A-9C The illustration shows an anomaly detection scenario in the implementation.

[0017] Figure 10 The illustrations depict computational systems suitable for implementing the enhanced anomaly detection techniques disclosed herein, including any architectures, processing, operating scenarios, and operation sequences shown in the figures and discussed in the detailed embodiments below. Detailed Implementation

[0018] The solution disclosed in this paper relates to block-based detection and prediction of low-probability behaviors (e.g., anomalous behaviors) in computing environments. An anomalous service receives log data from one or more edge nodes in a computing environment, where the edge nodes may be geographically distant from the service and / or co-located relative to the service. The log data includes a sequence of information indicating log messages generated by the nodes during runtime.

[0019] The anomaly service identifies the dominant pattern in the sequence of information representing non-anomaly blocks of log messages. The service extracts or otherwise ignores non-anomaly blocks from the log data to reveal anomaly (or low-probability) blocks that do not conform to the dominant pattern. The service then generates anomaly vectors based on these anomaly blocks and distributes them to the nodes.

[0020] One or more nodes in a computing environment receive anomaly vectors and use them to detect anomalous behavior in a stream of log messages. A given node generates a sequence of hash values ​​based on a sequence of log messages associated with events in the computing environment. The node also generates a sequence vector based on the hash values. The sequence vector includes, for example, numbers corresponding to a set of possible hash values ​​by position.

[0021] The node then performs a comparison between the sequence vector and the set of anomaly vectors to determine whether at least a portion of the hash values ​​indicates the occurrence of one or more anomalous events. If so, one or more actions can be taken to mitigate the risk of the problematic or anomalous events predicted by the comparison.

[0022] Returning to the reference anomaly service, the service can identify dominant patterns by finding and scoring latent patterns within the information sequence. The dominant pattern can then be selected based on the score. In some scenarios, the scoring function can be used to promote a subset of frequently occurring latent patterns within the information sequence relative to different subsets of latent patterns that occur less frequently within the information sequence. For each latent pattern, the scoring function determines the relative dominance of the latent pattern based on the description length of the information sequence when encoded using the compressed representation of the latent pattern.

[0023] In some implementations, the information sequence includes a sequence of log codes, and each log code may be a symbolic representation of a possible hash value. In this case, the log data will also include a codebook or dictionary that maps each unique log code to a different hash value in the set of possible hash values.

[0024] In various implementations, each exception vector includes a string of numbers that correspond to a set of possible log codes by position. The exception service sets the value of each number in the string to indicate the presence or absence of the corresponding log code among the possible log codes in the exception block represented by the exception vector.

[0025] Referring again to the client node, a given node generates a sequence vector by calculating a weighted value for each digit in the numeric string. The node calculates the weighted value by multiplying the weighting factor by a binary value. The binary value indicates whether the sequence of hash values ​​includes the hash value corresponding to the position of a given digit in the string, while the weighting factor indicates the relative importance of the log message to other log messages.

[0026] In some implementations, a node performs a comparison between a sequence vector and the set of anomalous vectors by applying a cosine similarity function to the inner dot product of the sequence vector and each anomalous vector in the set of anomalous vectors. The results can be compared to determine whether the sequence vector sufficiently matches one or more of the anomalous vectors, indicating an anomalous state of the computational environment.

[0027] It is understood that the anomaly detection techniques disclosed in this paper allow for more efficient detection of events that deviate from standards, normality, or expectation with less operational overhead. For example, log codes sent by client nodes to the anomaly service require less bandwidth than the full log statement itself, while distributing anomaly vectors to the edge allows anomalies to be detected at their source.

[0028] It can be further recognized that low-probability events—not just anomalous events—can also be predicted and / or detected. In such an implementation, the service receives log data from nodes in a computing environment, where the log data includes a sequence of information indicating log messages generated by the nodes. The service identifies a dominant pattern in the information sequence that represents high-probability blocks of log messages, and extracts high-probability blocks from the log data based on the dominant pattern, thereby revealing low-probability blocks that do not conform to the dominant pattern.

[0029] The service then generates low-probability vectors based on at least low-probability blocks and distributes the data to nodes to detect low-probability events. Nodes can then deploy these low-probability vectors (included in the data) for new log message streams and other such data.

[0030] Switch to the attached image. Figure 1 The diagram illustrates the operational environment 100 in implementation. The operational architecture 100 includes an exception service 110 that communicates with computing environments 101 and 111. Computing environment 101 includes one or more edge computing nodes represented by computing nodes 103 and 105. Computing environment 111 also includes one or more computing nodes represented by computing nodes 113 and 115. The exception service 110 receives log data from computing environments 101 and 111 and sends exception vectors to them. The exception service 110, along with computing environments 101 and 111, can... Figure 10 The computing system 1001 in the figure represents one or more server computers.

[0031] In operation, computing environments 101 and 111 employ log generation processing 200 to generate log data for anomaly service 110. Anomaly service 110 uses extraction processing 300 relative to the log data to extract anomaly blocks and generate anomaly vectors later supplied to computing environments 101 and 111. Computing environments 101 and 111 then employ detection processing 400 to detect, predict, or otherwise identify anomalous conditions in their respective environments.

[0032] Log generation processing 200 can be implemented in program instructions within the context of any software application, module, component, or other such program elements of computing environments 101 and 111. For example, log generation processing 200 can be implemented in the context of computing nodes 103 and 105 in computing environment 101 and in the context of computing nodes 113 and 115 in computing environment 111. Log generation processing 200 can also be implemented on any other computing element of computing environments 101 and 111, in addition to or in place of their respective computing nodes. The program instructions instruct(s)(s) the underlying physical or virtual computing system(s) to operate as described above, referencing... Figure 2 The steps are shown in the figure.

[0033] First, the log generation process 200 instructs the computing system (one or more) to identify a stream of log messages or statements relating to the operational state, application, service, process, or any other aspect of a given computing environment (step 201). Examples include log statements describing the start of a process, the end of a process, a process failure, the state of a database, the state of an application, or any other event that can be logged. Log messages can be identified by the same or different components that generated the messages.

[0034] Next, the log generation process 200 instructs one or more computing systems to generate an encoded version of the message, hereinafter referred to as the log code (step 203). For example, the log code could be a hash value generated by a hash function that takes a log statement as input and produces a hash value as output. One or more computing systems may modify the log message before hashing it, for example, by replacing the message's unique identifier, such as a server name or processing identifier, with a generic placeholder. This technique would allow hash functions to produce the same hash value for log messages that would be identical if they lacked their unique identifiers.

[0035] In some scenarios, log codes can be further encoded and / or compressed by replacing each of the various hash values ​​generated by a hash function with different symbols. For example, a hash function might output a string of numbers corresponding to a given log statement. The number string can be replaced with shorter alphanumeric symbols, thus reducing the size of the log code.

[0036] Finally, the log generation process 200 instructs one or more computing systems to send log data to an exception service (e.g., exception service 110) that includes log codes (step 205). The dictionary may include log data that maps various log codes to their corresponding log statements in their generic form. In scenarios where log codes are compressed representations of hash values, the dictionary can map the compressed representations to log statements and / or their corresponding hash values.

[0037] Figure 3 The extraction process 300 shown can be implemented in program instructions within the context of any software application, module, component, or other such program element including exception service 110. The program instructions instruct one or more underlying physical or virtual computing systems of exception service 110 to operate as follows, superficially referencing... Figure 3 The steps are shown in the figure.

[0038] Extraction process 300 begins by instructing one or more computing systems to receive log data from the computing nodes (step 301). The log data includes a sequence of information, such as log codes generated and timestamped by the computing nodes.

[0039] Next, the processing step 300 indicates that the computing system identifies a dominant pattern in the information sequence (step 303). Examples of dominant patterns include patterns of consecutive log codes that repeat within the information sequence or have a frequency greater than that of other less dominant patterns. For example, a snapshot of log data may include a thousand log codes, where blocks of three consecutive log codes repeat hundreds of times. This pattern can be considered a dominant pattern and therefore can be considered non-abnormal. That is, a dominant pattern may represent a log message pattern indicating a normal operating state rather than a guarantee of an alarm or other mitigated anomalous state.

[0040] Figure 3 Steps 303A-303D represent sub-processes for identifying dominant patterns in the message sequence, but other techniques are possible and within the scope of this disclosure. At step 303A, extraction process 300 instructs (one or more) the computing system to identify the root block within the message sequence. The root block may be, for example, the log code that appears most frequently (or most often) in the message sequence relative to any other log code.

[0041] Next, the extraction process 300 instructs (one or more) the computation system to grow at least one code or symbol for the root block at each position in the sequence where it occurs (step 303B) and calculates a Metadata Length (MDL) score for the new root block (step 303C). The MDL score is calculated by replacing the new root block at each position in the sequence with a symbol and then counting how much metadata is needed to describe the modified information sequence. Where the information sequence already includes symbols rather than hash values ​​themselves, the symbols replacing the new root block will replace the blocks of consecutive symbols that constitute the new root block.

[0042] Then, before expanding the root block, the extraction process 300 continues to compare the new value of the MDL score with the previous value of the MDL score (step 303D). If the MDL score is less than the previous MDL score, the subprocess proceeds to step 303B to grow the root block at each possible position in the sequence. If the root block can no longer be expanded, the process returns to step 303A to identify the next root block.

[0043] If the MDL score is not less than the previous MDL score, the process returns to step 303A to identify the next root block. If no more root blocks are to be identified and grown, the subprocess returns the resulting non-abnormal blocks to the main process. Alternatively, the subprocess may return to the main process after finding a threshold number of non-abnormal blocks, even if one or more root blocks may exist.

[0044] After identifying non-abnormal blocks, extraction process 300 instructs one or more computing systems to extract non-abnormal blocks from the information sequence, thereby revealing the remaining blocks that can be considered anomalous (step 305). Then, extraction process 300 instructs one or more computing systems to generate anomaly vectors based on the anomalous blocks (step 307) and distribute the vectors to computing environments to facilitate their detection of anomalous conditions.

[0045] In some implementations, the revealed blocks may be submitted to a labeling and review process to confirm that a given block represents an anomaly. The labeling and review process may be performed manually by a human operator, autonomously, or through a combination of manual and autonomous review. In other implementations, review is not required. When a review process is employed, the extraction process 300 is used to significantly reduce the amount of log statements requiring manual, autonomous, or a combination of manual and autonomous review. Optionally, the results of the review process may be provided as feedback to the extraction process 300 to further enhance its ability to identify anomalous blocks. Blocks revealed as not non-anomalies but rejected as non-anomalies during the review phase can be reclassified as non-anomalies and identified by the extraction process 300 for future use.

[0046] Anomalies can be identified by employing detection processing 400. Detection processing 400 can be implemented in program instructions within the context of any software application, module, component, or other such program elements of computing environments 101 and 111. For example, detection processing 400 can be implemented within the context of computing nodes 103 and 105 in computing environment 101 and in the context of computing nodes 113 and 115 in computing environment 111. Detection processing 400 can also be implemented on any other computing element of computing environments 101 and 111, in addition to or in place of their respective computing nodes. Program instructions instruct one or more underlying physical or virtual computing systems to operate as described above, referencing... Figure 4 The steps are shown in the figure.

[0047] The detection process 400 first instructs one or more computing systems to generate hash values ​​based on log messages generated during runtime (step 401). Then, the process instructs one or more computing systems to generate sequence vectors based on the hash values ​​(step 403). The sequence vector is a vectorized version of the sequence of hash values ​​generated from the log messages. In other words, the hash values ​​are presented in vectorized form. In some scenarios, the sequence vector can be generated directly from the log messages instead of hash values, or from a compressed representation of the hash values.

[0048] After generating the sequence vector, detection process 400 instructs one or more computing systems to compare the sequence vector with each anomaly vector in the set of anomaly vectors provided by the anomaly service (step 405). At step 407, detection process 400 instructs one or more computing systems to determine whether the sequence vector matches one or more of the anomaly vectors. If yes, then an anomaly is detected (step 409). If not, then the process continues to the next information sequence.

[0049] An exact match between the sequence vector and the anomaly vector is not required for identifying the anomaly. Rather, an approximate match may be sufficient to classify it as an anomaly. In some scenarios, a similarity function can be used to determine which anomaly vector fully matches the sequence vector. For example, a cosine similarity function can be applied to determine whether an anomaly vector fully matches the sequence vector. Furthermore, it is understood that the detection process 400 proceeds continuously as new log messages are generated.

[0050] Figure 5 This includes a simplified operation scenario 500, illustrated with an example of the operation sequence. In this operation, computing environment 101 generates log codes and sends the log codes from the log data to exception service 110. Exception service 110 also receives log data from computing environment 111. Exception service 110 processes the log codes in the log data to identify exception blocks and uses these exception blocks to generate exception vectors.

[0051] Anomaly vectors are sent by anomaly service 110 to computing environments 101 and 111. The corresponding computing environments can use the anomaly vectors to detect anomalies in the stream of log messages. In this way, various anomalous events, such as server failures and processing failures, can be detected and / or predicted.

[0052] Go back to reference Figure 1 Operation scenarios 120, 130, and 140 illustrate example implementations of log generation processing 200, extraction processing 300, and detection processing 400. It will be understood that while operation scenario 120 is illustrated with respect to computing environment 101, it can also occur in the context of computing environment 111. Similarly, operation scenario 140 is depicted with respect to computing environment 111, but it can also occur in the context of computing environment 101.

[0053] In operational scenario 120, one or more compute nodes in compute environment 101 generate a stream 121 of log messages. Logs in the stream are identified by their line numbers, such as line x-1, line x, and line x+1. A log generation process running in compute environment 101 performs encoding processing 123 on the log statements to generate log codes. For example, the process can anonymize each log statement by replacing server- or process-specific information with placeholder information. The anonymous log statements can then be hashed to produce hash values, and the hash values ​​can be further encoded into symbols, words, or other such compressed representations of the hash values.

[0054] As an example, once anonymized, the total number of k distinct log statements can be determined. Therefore, a deterministic encoding algorithm such as hashing will produce the same output code for each identical input statement. Thus, statement_1 will be encoded as code_1, statement_2 as code_2, and so on up to statement_k. The code is then further encoded into symbols or words. For example, code_1 maps to alpha, code_2 to bravo, and so on up to code_k, but any symbol is possible.

[0055] The encoded stream 125 is then sent from computing environment 101 to exception service 110. In this example, the encoded stream 125 comprises at least one contiguous block of symbols – echo, alpha, and bravo – tracing back to code_4, code_2, and code_1, respectively, which themselves trace back to anonymous statements_4, statement_1, and statement_2, respectively. The encoded stream may be sent along with other log data, such as dictionaries, which provide a mapping of symbols to codes and, possibly, to anonymous statements.

[0056] In operation scenario 130, exception service 110 receives log data generated by computing environment 101. Exception service 110 can also receive log data from computing environment 111. Log data 131 represents data that can be received from computing environment 101, computing environment 111, or both.

[0057] The log data includes the log code stream mined by the anomaly service 110 for anomaly block 133. For example, the anomaly service 110 identifies non-anomaly blocks that include log codes alpha, bravo, and delta. For illustrative purposes, it is assumed that the block occurs with a sufficient frequency (e.g., three times) to be considered a dominant block.

[0058] Removing one or more dominant blocks generates anomalous block 134 (echo) and anomalous block 135 (lima, zulu). Optionally, the revealed blocks can be submitted to the annotation and review process to confirm that the given blocks represent anomalous blocks. The annotation and review process can be performed manually by a human operator, autonomously, or through a combination of manual and autonomous review.

[0059] Anomaly service 110 continues to generate anomaly vectors based on the revealed anomaly blocks. Anomaly block 134 results in anomaly vector 138, while anomaly block 133 results in anomaly vector 139. Anomaly service 110 can then send the anomaly vectors to one or both of computing environments 101 and 111 for deployment in the advancement of anomaly detection / prediction.

[0060] In operation scenario 130, computing environment 111 has received an exception vector generated by exception service 110. Then, when a new log statement is generated, computing environment 111 is able to vectorize the log stream and compare it with the exception vector to determine whether there is an abnormal condition.

[0061] Log stream 141 represents a log stream generated by one or more elements in computing environment 111. A flow vector 143 is generated from log stream 141 so that it can be compared with anomaly vector 145. System state 147 can be determined based on the result of the comparison. If a match exists between flow vector 143 and one of anomaly vectors 145, then an abnormal state or condition is considered to exist. If not, then the system state is considered normal or at least non-abnormal.

[0062] Figure 6The illustration shows an operational scenario 600, a more detailed example of log generation processing. In this operation, the computing system generates a log message stream 601 related to the state of the information service. Each line of the log stream describes an event that occurred about one or more elements of the service. For example, line 1 indicates that a new process has been initiated. This process is identified by the process ID, and the element on which the process was initiated is identified by the host ID and client ID. Line 2 describes the start of a process on the same computing element that differs from line 1.

[0063] Then, in line 3, different events are described. The process described in line 2 is restarted by a combination of the components described in line 2. Line 4 describes the same process that fails due to a network disconnection. Finally, line 5 describes the restart failure of the same process described in line 4.

[0064] It can be recognized that all five exemplary log messages belong to the same host / client combination, but refer to two different processes. The host identifier uniquely identifies the machine from among other machines in the computing environment, while the client identifier uniquely identifies the client from among other clients on the machine. The process identifier also uniquely identifies the process associated with other processes on the machine. However, the identifiers can make it difficult to detect and / or predict anomalous conditions from log statements, as they make each statement distinct from every other statement. (As mentioned herein, the term "machine" can mean a physical computer, virtual machine, container, or any other type of computing resource, variant, or combination thereof.)

[0065] To mitigate these effects, the log generation process (e.g., 200) removes the processor, machine, and client identifiers from copies of the log messages and replaces them with placeholder information. This step can be performed on all log messages generated by all other machines in a given computing environment, thus clearing the aggregated stream of log messages of such distinguishing identifiers. The original version of the log message with complete identifiers can be retained for other purposes.

[0066] The modified log stream 603 provides an example where the processing identifier in lines 1-5 has been replaced with the acronym "PH," but any word or symbol is possible. Similarly, combinations of host and client identifiers have been replaced with the same acronym. In general, anonymizing or flattening log statements across all machines can improve their usefulness for anomaly detection and prediction.

[0067] The modified log stream 603 is then fed to the hash function 605. The hash function, which can be implemented in program instructions on a computing system, takes each line as input to generate a corresponding hash value. Because hash functions are deterministic, the same input always produces the same output. Combined with the modified log messages, this feature can produce the same hash value for multiple log statements that are initially different but identical after modification. For example, lines 1 and 2 of the original log stream differ because of their processing IDs, but they are the same after their processing IDs (and machine / client IDs) are replaced with placeholders. Therefore, the hash value produced by hash function 605 is the same (e.g., hash_1). The hash value of line 5 produces the same result, but the hash values ​​of lines 3 and 4 are different (hash_2, hash_3) because their corresponding log statements differ from the others after the placeholder replacements.

[0068] The encoded log stream 607 generated by hash function 605 can then be sent to the exception service (e.g., 110). The encoded log stream 607 includes hash values ​​in encoded / compressed forms (e.g., hash_1, hash_2) relative to their actual values. A dictionary for decoding / decompressing symbols can also be provided to the exception service. Optionally, the actual hash value itself can be sent to the exception service.

[0069] Figure 7 The illustration shows a more detailed example of an operational scenario 700 of the exception block extraction process (e.g., 300) in the implementation. In this operation, the exception service receives one or more encoded log streams, represented by encoded log stream 701, from one or more computing environments. For illustrative purposes, the encoded log stream 701 comprises five lines, each containing information such as... Figure 6 The log code representing the symbolic / encoded hash value under discussion.

[0070] The exception service feeds log codes to a block extraction function 703, which can be implemented in program instructions on one or more computing systems of the service. The block extraction function 703 examines the log code sequence to identify consecutive blocks of code that appear more frequently than other codes. For example, for illustrative purposes, suppose the function finds a candidate block 705 that appears 200 times in the examined log code sequence. For comparison purposes, a candidate block 707 is also found, but it appears only once in the log code sequence.

[0071] The candidate blocks and their corresponding counts are then fed to annotation function 709, which may also be implemented in program instructions on the serving computing system(s). Annotation function 709 determines whether a given candidate block is classified as a non-abnormal block or an anomalous block based at least in part on its count. Candidate block 705 is classified as a non-abnormal block 711 and is therefore removed from the sequence. Candidate block 707 is classified as an anomalous block 712 and is fed as input to vectorization function 713, which vectorizes the log codes(s) contained in the anomalous block(s).

[0072] To vectorize exception block 712, vectorization function 713 identifies which log codes are included in the block. Vectorization function 713 can also be implemented in program instructions on the service's computing system(s). In this example, the block includes only one log code (hash_3). Vectorization function 713 then feeds values ​​to the positions in exception vector 715 corresponding to the log codes represented in the block. For example, the third position from right to left in exception vector 715 is fed the value "1" to indicate that exception block 712 includes the log code "hash_3". For illustrative purposes, suppose the exception block also includes the log code "hash_5", then the fifth position in the exception vector would be fed the value "1". The remaining positions in the vector are fed the value "0" to indicate that no corresponding log code exists in the block.

[0073] Figure 8 The illustration shows an example scenario 800 of the block extraction process in the implementation. In example scenario 800, the processing table 801 is processed step by step from top to bottom in the order of operations to identify abnormal blocks in the log sequence.

[0074] The first column of processing table 801 describes the abbreviated replacements of letters for the hash codes. These replacements are not actually performed but are shown for illustrative purposes only. Log codes hash_1 are represented by A, hash_2 by B, and so on. For illustrative purposes, it is assumed that there are twenty-six unique log codes in the encoded log stream, but any number of log codes is possible.

[0075] Next, Table 801 indicates the current root block being evaluated, along with characters and / or symbols in the log sequence. Initially, the sequence is analyzed to find the most frequent log codes to be set as the initial value for the root block (r-block). Using A as the root block, the root block's Metadata Length (MDL) score is initially twenty-three.

[0076] Therefore, the evaluation continues by growing a symbol for root block A at all its positions in the sequence. Thus, the first potential root block (p-block) is found to be AB. The continuous symbol AB is replaced by symbol α at all its positions in the sequence, and the MDL score is calculated to be nineteen. Because the MDL score of the potential block is less than that of the root block, the evaluation continues by replacing the root block with the potential block.

[0077] Therefore, the root block becomes α, and the evaluation attempts to grow the root block at all positions in the sequence. In cases where multiple potential blocks can be grown from the root block, the evaluation selects the most frequent potential block. For example, α can be grown by adding C at four positions and D at three positions. Therefore, the evaluation proceeds to αC as the next potential block, and restores α to its original character set AB at its other positions. The potential block αC is then replaced with the symbol λ at all positions in the sequence, and the MDL score of αC is calculated. Because the MDL score of the potential block λ is less than the MDL score of the root block α, the evaluation continues to replace the root block with the potential block.

[0078] Then, the evaluation attempts to grow the root block at all positions. However, the root block λ cannot grow at any position in the sequence. Therefore, the root block λ is considered a non-abnormal block, and its characters or values ​​can be removed from the sequence. Thus, the root block is restored to its previous value α, and the evaluation starts again, but ABC is replaced with λ at all positions.

[0079] With the root block set to α, the evaluation attempts to grow the root block at all its positions and identify new potential blocks δ = αD or ABD. The potential block ABD is then replaced with the symbol δ at all its positions in the sequence, and the MDL score of 15 is calculated. Because the MDL score of the potential block δ is less than the MDL score of the root block α, the evaluation continues to replace the root block with the potential block.

[0080] Then, the evaluation attempts to grow the root block δ at all its locations are performed, and after determining that it cannot grow, the next root block is identified. The next root block is Y, with an MDL score of 14. By adding Z, Y can grow at only one location, and the MDL score of the potential block YZ is 16. Because 16 is greater than 14, YZ fails as a potential root block and is therefore revealed as an anomalous block.

[0081] The exception blocks can then be vectorized and distributed to clients to detect and / or predict anomalous operational states. Figures 9A-9C The illustration depicts a scenario ranging from the generation of exception vectors in an abnormal service to their deployment on the client. Figure 9AIn the first part of scenario 900A shown, the exception service has identified exception block 901 from the log code stream generated by one or more clients in one or more computing environments. Exception block 901 includes blocks 903, 905, and 907. Block 903 includes two log codes represented by hash_1 and hash_5. Block 905 includes one log code represented only by hash_3. Block 907 includes two log codes represented by hash_3 and hash_n.

[0082] The leftmost column of vector table 909 lists all n possible hash codes from hash_1 to hash-n. The top row identifies anomalous blocks from the first block b_1 to the kth block b_k. Each cell defined by a given hash code / block combination contains a value indicating whether the block includes a hash code. Zero indicates that the block does not include a hash code, while one indicates that the block does include a hash code. Regarding anomalous block 901, block 603 is defined in vector table 909 at the cells defined by hash_1 and b_1, and at the cells defined by hash_5 and b_1. The remaining cells in column b_1 have zero values ​​because block 903 does not contain any other hash codes.

[0083] The vector values ​​of exception blocks b_2 and b_3 are filled in the same way. That is, block 905 in vector table 909 is defined by a 1 at the cell defined by hash_3 and b_2 and a zero at all other locations. Block 907 is defined in vector table 909 by a 1 at the cell defined by hash_3 and b_3 and by a 1 at the cell defined by hash_n and b_3. The remaining cells in column b_3 have zero values ​​because block 907 does not contain any other hash codes.

[0084] The resulting anomaly vector 911 is a string of binary values ​​from right to left corresponding to the positions in the cells of each block in vector table 909. For example, the anomaly vector 913 corresponding to block 903 consists of the string 0...0010001. The "1" in the rightmost position corresponds to the value in the cell defined by hash_1 and b_1, while the "1" in the fifth position from the right corresponds to the value in the cell defined by hash_5 and b_1. All other values ​​in anomaly vector 913 are "0".

[0085] The exception vector 915 corresponding to block 905 follows the same convention and includes the string 0...0000100. The "1" in the third position from the right corresponds to the value in the cell defined by hash_3 and b_2, while all other values ​​are zero.

[0086] The anomaly vector 917 corresponding to block 907 consists of the string 1...0000100. The leftmost "1" corresponds to the value in the cell defined by hash_n and b_3, while the third "1" from the right corresponds to the value in the cell defined by hash_3 and b_3. All other values ​​in anomaly vector 917 are "0".

[0087] Anomaly services distribute vectors to one or more computing environments to facilitate anomaly detection and prediction, such as... Figures 9A-9C The second part of the described operational scenario is shown in 900B.

[0088] Figure 9B This includes log stream 921, representing a stream of log codes generated by the client's computing system. In this example, the log codes are hash values, which the client inputs to a vectorization function 923. The vectorization function vectorizes log stream 921 to produce vector stream 925. Figure 9A The vectorization process shown is similar; each log code can be vectorized by indicating a "1" at the position in the vector corresponding to the hash value represented by the log code.

[0089] As an example, the set of log codes in the middle of log stream 921 is shown in bold, and their corresponding vectors are provided in vector stream 925. According to this convention, the log code represented by hash_4 is transformed into a vector with a "1" at the fourth position from the right and zeros at all other positions. Therefore, the log code represented by hash_3 is transformed into a vector with a "1" at the third position from the right and zeros at all other positions. The log code represented by hash_2 is transformed into a vector with a "1" at the second position from the right and zeros at all other positions. Finally, the log code represented by hash_n is transformed into a vector with a "1" at the nth position from the right and zeros at all other positions.

[0090] The client inputs vector stream 925 into weighting function 927. Weighting function 927 applies weighted vectors 928 to each individual vector in vector stream 925. Weighted vector 928 includes weight values ​​between 0 and 1 at each position in the vector, corresponding to the position in the vector in vector stream 925. For example, a "1" at a position in weighted vector 928 indicates that the corresponding position in vector stream 925 should be assigned its full value, while a ".5" at a position in weighted vector 928 indicates that the corresponding position in vector stream 925 is at half its full value.

[0091] The weighting function 927 produces a weighted vector stream 929, which is the product of each position in the weighted vector 928 and its corresponding position in each vector of the vector stream 925. As an example, the intermediate vectors in the vector stream 925 and the weighted vector stream 929 are shown in bold to highlight the product of the weight value (.7) given to hash_3 multiplied by the value (1) at position hash_3 in the intermediate vector. As another example, the second vector at the bottom of the weighted vector stream 929 from the vector stream 929 is shown in bold to highlight the zero weight given to hash_2 in the weighted vector 928.

[0092] at last, Figure 9C The diagram shows... Figures 9A-9C The last part of the operational scenario shown is 900C. In Figure 9C In this context, the exception vector 911 generated by the exception service has been distributed to clients in their respective computing environments. Each client generates its own vector stream, represented by a weighted vector stream 929.

[0093] The comparison function 931 receives anomaly vectors 911 and a weighted vector stream 929 as input, and performs a comparison computation on each vector in the weighted vector stream 929 for each vector in the anomaly vectors 911. The comparison function can be, for example, a cosine similarity function, which evaluates each anomaly vector multiplied by the dot product of each vector in the vector stream. The comparison function then analyzes the computation to determine whether one of the weighted anomaly vectors 911 includes a sufficient match with one or more of the anomaly vectors 911. If so, then an anomaly is considered to exist or predicted. If not, then a normal state exists.

[0094] Figure 10 The illustration depicts a computing system 1001, which represents any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein may be implemented. Examples of computing system 1001 include, but are not limited to, server computers, routers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, physical or virtual router, container, and any variations or combinations thereof.

[0095] The computing system 1001 can be implemented as a single device, system, or apparatus, or it can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing system 1001 includes, but is not limited to, a processing system 1002, a storage system 1003, software 1005, a communication interface system 1007, and a user interface system 1009 (optional). The processing system 1002 is operatively coupled to the storage system 1003, the communication interface system 1007, and the user interface system 1009.

[0096] Processing system 1002 loads and executes software 1005 from storage system 1003. Software 1005 includes and implements exception handling 1006, which represents the processing discussed with respect to the foregoing figures, including log generation processing 200, extraction processing 300, and detection processing 400. When executed by processing system 1002 to provide block-based exception detection and prediction, software 1005 instructs processing system 1002 to operate as described herein for the various processing, operational scenarios, and sequences discussed at least in the foregoing embodiments. Computing system 1001 may optionally include additional devices, features, or functions not discussed for brevity.

[0097] Still referencing Figure 10 The processing system 1002 may include a microprocessor and other circuitry for retrieving and executing software 1005 from the storage system 1003. The processing system 1002 may be implemented within a single processing device, or it may be distributed across multiple processing devices or subsystems that collaboratively execute program instructions. Examples of the processing system 1002 include general-purpose central processing units, graphics processing units, dedicated processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof.

[0098] Storage system 1003 may include any computer-readable storage medium that can be read by processing system 1002 and is capable of storing software 1005. Storage system 1003 may include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, optical media, flash memory, virtual and non-virtual memory, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other suitable storage medium. In any case, computer-readable storage media are not for transmitting signals.

[0099] In addition to computer-readable storage media, in some embodiments, storage system 1003 may also include computer-readable communication media on which at least some of the software 1005 can be transmitted internally or externally. Storage system 1003 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems that are co-located or distributed relative to each other. Storage system 1003 may include additional elements, such as controllers, capable of communicating with processing system 1002 or possibly other systems.

[0100] Software 1005 (including exception handling 1006) can be implemented in program instructions and, among other functions, can instruct processing system 1002 to operate as described herein in the various operational scenarios, sequences, and processes illustrated herein. For example, software 1005 may include program instructions for implementing log generation processing, exception block extraction processing, and exception detection processing as described herein.

[0101] Specifically, program instructions may include various components or modules that cooperate or otherwise interact to perform the various processing and operational scenarios described herein. These components or modules may be implemented in compiled or interpreted instructions, or in some other variation or combination of instructions. They may be executed synchronously or asynchronously, serially or in parallel, in a single-threaded or multi-threaded environment, or according to any other suitable execution paradigm, variation, or combination thereof. Software 1005 may include additional processing, programs, or components, such as operating system software, virtualization software, or other application software. Software 1005 may also include firmware or some other form of machine-readable processing instructions executable by processing system 1002.

[0102] Generally, when loaded into and executed by processing system 1002, software 1005 can transform a suitable apparatus, system, or device (represented by computing system 1001) from a general-purpose computing system to a specialized computing system tailored to provide block-based exception handling as described herein. In practice, the coding software 1005 on storage system 1003 can transform the physical structure of storage system 1003. Specific transformations of the physical structure can depend on various factors in the different embodiments described herein. Examples of these factors may include, but are not limited to, the technology used to implement storage system 1003 and whether the computer storage medium is characterized as primary or secondary storage, among other factors.

[0103] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then the software 1005 can transform the physical state of the semiconductor memory as program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. Similar transformations can occur for magnetic or optical media. Other transformations of the physical medium are possible without departing from the scope of this specification; the foregoing examples are provided merely for the convenience of this discussion.

[0104] The communication interface system 1007 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that together allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry systems. The connections and devices may communicate via a communication medium to exchange communication with other computing systems or a network of systems, such as metal, glass, air, or any other suitable communication medium. The media, connections, and devices mentioned above are well-known and do not require detailed discussion here.

[0105] Communication between computing system 1001 and other computing systems (not shown) may occur on one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The communication networks and protocols mentioned above are well-known and do not require detailed discussion here.

[0106] As will be appreciated by those skilled in the art, aspects of the present invention can be implemented as systems, methods, or computer program products. Thus, aspects of the invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which can generally be referred to as “circuit,” “module,” or “system.” Furthermore, aspects of the invention can take the form of computer program products embodied in one or more computer-readable media having computer-readable program code contained thereon.

[0107] The included description and accompanying drawings depict specific embodiments that teach those skilled in the art how to make and use the best mode. Some conventional aspects have been simplified or omitted for the purpose of teaching the principles of the invention. Those skilled in the art will recognize variations of these embodiments that fall within the scope of this disclosure. They will also recognize that the foregoing features can be combined in various ways to form multiple embodiments. Therefore, the invention is not limited to the specific embodiments described above, but is limited only by the claims and their equivalents.

Claims

1. A computing device, comprising: One or more computer-readable storage media; One or more processors operatively coupled to the one or more computer-readable storage media; as well as Program instructions for detecting anomalies, which, when executed by the one or more processors, instruct the computing device to at least: Receive an anomaly vector set based on a first log message from multiple edge nodes from a computing device that includes an anomaly service, the anomaly vector set indicating low-probability blocks in the first log message that do not match the dominant pattern in the first log message; A sequence of hash values ​​is generated based at least on a sequence of second log messages associated with events in the computing device; Generate a sequence vector based on the hash value, wherein the sequence vector comprises a string of numbers corresponding to a set of possible hash values ​​by position; as well as Perform a comparison between the sequence vector and the set of anomaly vectors to determine whether at least a portion of the hash values ​​indicates the occurrence of one or more anomalous events.

2. The computing device as claimed in claim 1, wherein, In order to generate the sequence vector based on the hash value sequence, the program instructions instruct the computing device to calculate a weighted value for each number in the string of numbers.

3. The computing device as claimed in claim 2, wherein, In order to calculate a weighted value for each number in the string of numbers, the program instructions instruct the computing device to multiply the weighting factor by a binary value to obtain the weighted value.

4. The computing device as claimed in claim 3, wherein, The binary value indicates whether the hash value sequence includes a hash value corresponding to the position of a given number in the string.

5. The computing device as claimed in claim 4, wherein, The weighting factor indicates the relative importance of the log message in detecting the one or more anomalous events.

6. The computing device as claimed in claim 5, wherein, In order to perform the comparison between the sequence vector and the set of anomalous vectors, the program instructions instruct the computing device to apply a cosine similarity function to the inner dot product of the sequence vector and each anomalous vector in the set of anomalous vectors.

7. A method for detecting anomalies, comprising: The first computing device receives a set of anomaly vectors based on first log messages from multiple edge nodes from a second computing device that includes anomaly services. The set of anomaly vectors indicates low-probability blocks in the first log messages that do not match the dominant pattern in the first log messages. The first computing device identifies a sequence of second log messages associated with events in the first computing device; The first computing device applies a hash function to a sequence of second log messages to generate a sequence of hash values; A first computing device generates a sequence vector based on a hash value, wherein the sequence vector comprises a string of numbers corresponding to a set of possible hash values ​​by position; as well as The first computing device performs a comparison between the sequence vector and the set of anomaly vectors to determine whether at least a portion of the hash values ​​indicates the occurrence of one or more anomalous events.

8. The method of claim 7, wherein, Generating the sequence vector based on the hash value sequence includes calculating a weighted value for each number in the string of numbers.

9. The method of claim 8, wherein, Calculating a weighted value for each number in the string of numbers involves multiplying a weighting factor by a binary value to obtain the weighted value.

10. The method of claim 9, wherein, The binary value indicates whether the hash value sequence includes a hash value corresponding to the position of a given number in the string.

11. The method of claim 10, wherein, The weighting factor indicates the relative importance of the log message in detecting the one or more anomalous events.

12. The method of claim 11, wherein, Performing a comparison between the sequence vector and the set of anomalous vectors includes applying a cosine similarity function to the inner dot product of the sequence vector and each anomalous vector in the set of anomalous vectors.

13. A computing device, comprising: One or more computer-readable storage media; One or more processors operatively coupled to the one or more computer-readable storage media; as well as Program instructions for detecting anomalies, which, when executed by the one or more processors, instruct the computing device to at least: Receive an anomaly vector set based on a first log message from multiple edge nodes from a computing device that includes an anomaly service, the anomaly vector set indicating low-probability blocks in the first log message that do not match the dominant pattern in the first log message; Identify a sequence of second log messages associated with events in the computing device; Generate a compressed message sequence corresponding to the sequence of the second log message; A sequence vector is generated based on the compressed message sequence, wherein the sequence vector includes a string of numbers corresponding to a possible set of compressed messages by position; as well as Perform a comparison between the sequence vector and the set of anomaly vectors to determine whether at least a portion of the sequence of the second log message indicates the occurrence of one or more anomalous events.

14. The computing device of claim 13, wherein, The compressed message sequence includes a hash value sequence, and the possible compressed message set includes a possible hash value set.

15. The computing device of claim 14, wherein, In order to generate the compressed message sequence, the program instructions instruct the computing device to apply a hash function to the log message sequence to produce the hash value sequence.

16. The computing device of claim 15, wherein, In order to generate the sequence vector based on the hash value sequence, the program instructions instruct the computing device to calculate a weighted value for each number in the string of numbers.

17. The computing device of claim 16, wherein, In order to calculate a weighted value for each number in the string of numbers, the program instructions instruct the computing device to multiply the weighting factor by a binary value to obtain the weighted value.

18. The computing device of claim 17, wherein, The binary value indicates whether the hash value sequence includes a hash value corresponding to the position of a given number in the string.

19. The computing device of claim 18, wherein, The weighting factor indicates the relative importance of the log message in detecting the one or more anomalous events.

20. The computing device of claim 19, wherein, In order to perform the comparison between the sequence vector and the set of anomalous vectors, the program instructions instruct the computing device to apply a cosine similarity function to the inner dot product of the sequence vector and each anomalous vector in the set of anomalous vectors.

Images