Anomaly identification method for distributed architecture and related apparatus

Abstract

The present disclosure relates to an anomaly identification method and related apparatus for a distributed architecture. An anomaly identification method for a distributed architecture includes converting text of content of a first anomaly collected in a first component of the distributed architecture into a SimHash value; determining a minimum Hamming distance among Hamming distances between the SimHash value of the first anomaly and SimHash values of each anomaly in an anomaly inventory table, the anomaly inventory table being used to maintain non-repeating anomalies that have occurred in the distributed architecture; converting the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly; correcting the probability based on at least one of a length of the text of content of the first anomaly and an instability of the first component; and identifying whether the first anomaly is a new anomaly based on the corrected probability.

Description

Technical Field

[0001] This disclosure relates to the field of anomaly detection technology, and more specifically, to an anomaly detection method, apparatus, computing device, and non-transient storage medium for distributed architectures. Background Technology

[0002] For stability and scalability reasons, many platforms adopt a distributed architecture. Each component in a distributed architecture has its own log information, resulting in tens of thousands of anomaly logs being recorded daily. Taking an airline's airport data sharing platform as an example, most of the anomalies recorded daily are caused by systems of external entities such as the airport, including external platform interface anomalies, network anomalies, and external data anomalies. The remainder are various anomalies on the platform's internal distributed application nodes. Because it is difficult to identify valuable information from the massive amount of anomaly data daily using manual methods, traditional platform maintenance simply records logs, only tracing the logs to find the cause when serious problems occur. This leads to delays in discovering and resolving problems and makes it difficult to detect potential platform vulnerabilities. Summary of the Invention

[0003] According to a first aspect of this disclosure, an anomaly identification method for a distributed architecture is provided, comprising: converting the text of the content of a first anomaly collected in a first component of the distributed architecture into a SimHash value; determining a minimum Hamming distance among Hamming distances between the SimHash value of the first anomaly and the SimHash value of each anomaly in an anomaly list table, the anomaly list table being used to maintain non-repeating anomalies that have occurred in the distributed architecture; converting the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly; correcting the probability based on at least one of the length of the text of the content of the first anomaly and the instability of the first component; and identifying whether the first anomaly is a new anomaly based on the corrected probability.

[0004] In some embodiments, the anomaly identification method further includes: in response to identifying the first anomaly as a new anomaly, adding a record of the first anomaly to the anomaly list table.

[0005] In some embodiments, the anomaly identification method further includes: when the corrected probability is less than a preset probability threshold, identifying the first anomaly as a second anomaly in the anomaly list table that has the minimum Hamming distance from the first anomaly; when the corrected probability is not less than the preset probability threshold, identifying the first anomaly as a new anomaly and adding a record of the first anomaly to the anomaly list table, the record including the descriptive attributes of the first anomaly.

[0006] In some embodiments, the anomaly identification method further includes: when the corrected probability is less than the preset probability threshold, adding a record of the first anomaly to an anomaly event table, the record including the primary key of the second anomaly in the anomaly list table and the instantaneous attribute of the first anomaly, the anomaly event table being used to record each anomaly that has occurred in the distributed architecture; when the corrected probability is not less than the preset probability threshold, adding a record of the first anomaly to the anomaly event table, the record including the primary key of the first anomaly in the anomaly list table and the instantaneous attribute of the first anomaly.

[0007] In some embodiments, converting the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly includes one of the following: mapping the minimum Hamming distance to the interval [0,1] and using the mapping result as the probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result increases as the minimum Hamming distance increases; or mapping the difference or ratio between the minimum Hamming distance and a preset Hamming distance threshold to the interval [0,1] and using the mapping result as the probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result increases as the difference or ratio increases.

[0008] In some embodiments, correcting the probability based on the length of the text of the first abnormal content includes one of the following: mapping the length of the text of the first abnormal content to the interval [0,1], and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases; or mapping the difference or ratio between the length of the text of the first abnormal content and a preset text length threshold to the interval [0,1], and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the difference or ratio decreases.

[0009] In some embodiments, correcting the probability based on the length of the text of the first abnormal content includes: when the length of the text of the first abnormal content is not less than a preset text length threshold, not correcting the probability based on the length of the text of the first abnormal content; when the length of the text of the first abnormal content is less than the preset text length threshold: mapping the difference or ratio between the length of the text of the first abnormal content and the preset text length threshold to the [0,1] interval, and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the difference or ratio decreases; or mapping the length of the text of the first abnormal content to the [0,1] interval, and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases.

[0010] In some embodiments, correcting the probability based on the instability of the first component includes: mapping the instability of the first component to the interval [0,1], and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result increases as the instability of the first component decreases.

[0011] In some embodiments, correcting the probability based on the instability of the first component includes: when the instability of the first component is not greater than a preset instability threshold, not correcting the probability based on the instability of the first component; when the instability of the first component is greater than the preset instability threshold, mapping the instability of the first component to the [0,1] interval, and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is larger when the instability of the first component is smaller.

[0012] In some embodiments, mapping the instability of the first component to the [0,1] interval includes: allocating a ticket box with a maximum number of tickets of N to the first component and initializing the ticket box by filling it with tickets, wherein the remaining number of tickets in the ticket box is set to X2, adding δ tickets to the ticket box every preset time period when the ticket box is not full, and removing (δ+(N-X2)) tickets from the ticket box whenever the first component malfunctions, and wherein the instability of the first component is represented as...

[0013] In some embodiments, the modified probability P is expressed as:

[0014]

[0015] Where X1 is the minimum Hamming distance, and D tX1 is the preset Hamming distance threshold, X2 is the remaining number of votes in the first component's ballot box, N is the maximum number of votes in the first component's ballot box, X3 is the length of the text of the first abnormal content, and L is the maximum number of votes in the first component's ballot box. t W1 is the preset text length threshold, and W2 and W3 are weight adjustment parameters in the range [0,1].

[0016] According to a second aspect of this disclosure, an anomaly identification apparatus for a distributed architecture is provided, comprising a processing module, a correction module, and an identification module. The processing module is configured to: convert the text of the content of a first anomaly collected in a first component of the distributed architecture into a SimHash value; determine the minimum Hamming distance among the Hamming distances between the SimHash value of the first anomaly and the SimHash values ​​of each anomaly in an anomaly list table, the anomaly list table being used to maintain non-repeating anomalies that have occurred in the distributed architecture; and convert the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly. The correction module is configured to correct the probability based on at least one of the length of the text of the content of the first anomaly and the instability of the first component. The identification module is configured to identify whether the first anomaly is a new anomaly based on the corrected probability.

[0017] According to a third aspect of this disclosure, a computing device for anomaly identification in a distributed architecture is provided, comprising: one or more processors; and a memory storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform anomaly identification method for a distributed architecture according to any embodiment of the first aspect of this disclosure.

[0018] According to a fourth aspect of this disclosure, a non-transient storage medium having computer-executable instructions stored thereon is provided, which, when executed by a computer, cause the computer to perform an anomaly identification method for a distributed architecture according to any embodiment of the first aspect of this disclosure.

[0019] Other features and advantages of this disclosure will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0021] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0022] Figure 1This is a flowchart of an anomaly identification method for a distributed architecture according to some embodiments of the present disclosure;

[0023] Figure 2 These are exception listing tables and exception event tables for distributed architectures according to some embodiments of this disclosure;

[0024] Figure 3 This is a schematic block diagram of an anomaly detection device for a distributed architecture according to some embodiments of the present disclosure;

[0025] Figure 4 This is a schematic block diagram of a computing device for anomaly identification in a distributed architecture according to some embodiments of the present disclosure.

[0026] Note that in the embodiments described below, the same reference numerals are sometimes used across different figures to denote the same parts or parts having the same function, and repeated descriptions are omitted. In this specification, similar reference numerals and letters are used to denote similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0027] For ease of understanding, the positions, dimensions, and extents of the structures shown in the accompanying drawings and other materials may not represent actual positions, dimensions, and extents. Therefore, the disclosed invention is not limited to the positions, dimensions, and extents disclosed in the accompanying drawings and other materials. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components. Detailed Implementation

[0028] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0029] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. Those skilled in the art will understand that they merely illustrate exemplary ways that can be used to implement this disclosure, and are not exhaustive.

[0030] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0031] Exceptions in distributed architectures can include, but are not limited to, program exceptions, message queue exceptions, network exceptions, database exceptions, hardware exceptions, data exceptions, message queue exceptions, and custom exceptions. Program exceptions often include stack traces and may include runtime errors in microservices or data processing scripts, resource alerts such as thread counts, etc. Due to dynamic proxies or dynamic parameters in their content, the text of the same type of exception may not be exactly the same each time it occurs, making simple equality rules unsuitable for exception identification. However, applying regular expressions to each exception would incur extremely high development and maintenance costs in a large distributed architecture, making it difficult to implement.

[0032] To address this, this disclosure provides an improved anomaly detection method for distributed architectures, which can perform anomaly detection quickly, reliably, and even in real-time. The anomaly detection methods according to various embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0033] Figure 1 An anomaly identification method 100 (hereinafter referred to as method 100) for a distributed architecture is illustrated according to some embodiments of the present disclosure.

[0034] like Figure 1 As shown, method 100 includes: at step S102, converting the text of the content of the first exception collected in the first component of the distributed architecture into a SimHash value; at step S104, determining the minimum Hamming distance among the Hamming distances between the SimHash value of the first exception and the SimHash value of each exception in the exception list table.

[0035] The distributed architecture comprises multiple components, where each exception occurring in each component can be collected via a corresponding channel. The first component mentioned here can be any component in the distributed architecture, and the first exception mentioned here can be any exception that can occur in the first component.

[0036] An exception list can be used to maintain a unique list of exceptions that have occurred in a distributed architecture. Figure 2 The table dim_meta_error_list shown is a non-restricted example of an exception list table. Figure 2 As shown, the exception list table (dim_meta_error_list) records multiple data entries for each exception, including a primary key (or exception list primary key), exception content (text), tag, component containing the exception, creation time, SimHash value of the exception content (text), and length of the exception content (text). The exception list table can be used as a dimension table to store exception identification results, and the exceptions within it are unique.

[0037] The SimHash algorithm is a fast hashing method commonly used for calculating text similarity and deduplication. The SimHash value it produces is binary. The number of bits that differ between corresponding bits of two SimHash values ​​is called the Hamming distance between them. The Hamming distance is a natural number, and when the Hamming distance is zero, it indicates that the two SimHash values ​​are essentially equal, while a larger Hamming distance indicates a greater difference between them. By determining the minimum Hamming distance between the SimHash value of the first anomaly and the SimHash value of every anomaly in the anomaly list, we can find the anomaly in the anomaly list that is most similar to the first anomaly (referred to as the second anomaly in this paper).

[0038] Continue to refer to Figure 1 Method 100 further includes: at step S106, converting the minimum Hamming distance determined at step S104 into a probability indicating that the first anomaly is a new anomaly.

[0039] In some embodiments, step S106 may include mapping the minimum Hamming distance determined in step S104 to the interval [0,1], and using the mapping result as a probability indicating that the first anomaly is a new anomaly. Any suitable mapping may be used, as long as it is configured such that the mapping result increases as the minimum Hamming distance increases.

[0040] In some embodiments, step S106 may include mapping the difference between the minimum Hamming distance determined in step S104 and a preset Hamming distance threshold to the interval [0,1], and using the mapping result as the probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result increases as the difference increases. For example, the preset Hamming distance threshold may be a value set empirically, and may have different suitable values ​​in different application scenarios. In some examples, the preset Hamming distance threshold may be selected as 5, or 3, etc. It should also be noted that when the minimum Hamming distance is greater than the preset Hamming distance threshold, the difference between the minimum Hamming distance and the preset Hamming distance threshold is positive; when the minimum Hamming distance is less than the preset Hamming distance threshold, the difference between the minimum Hamming distance and the preset Hamming distance threshold is negative. In some embodiments, the mapping in step S106 may be implemented using the sigmoid function, so the probability (P0) indicating that the first anomaly is a new anomaly may be expressed as, for example, P0 = Sigmoid(X1 - D) t ), where X1 is the minimum Hamming distance, D tThis is a preset Hamming distance threshold. Of course, other functions used for mapping to the [0,1] interval are also feasible, as long as the mapping result increases as the difference between the minimum Hamming distance and the preset Hamming distance threshold increases. The more the minimum Hamming distance is smaller than the preset Hamming distance threshold, the more similar the first anomaly is to the second anomaly in the anomaly list, thus the lower the probability that the first anomaly is a new anomaly. Conversely, the more the minimum Hamming distance is larger than the preset Hamming distance threshold, the less similar the first anomaly is to the second anomaly in the anomaly list, thus the higher the probability that the first anomaly is a new anomaly.

[0041] In other embodiments, step S106 may also include mapping the ratio between the minimum Hamming distance determined in step S104 and a preset Hamming distance threshold to the [0,1] interval, and using the mapping result as the probability indicating that the first anomaly is a new anomaly. Any suitable mapping can be used here, as long as the mapping is configured such that the mapping result is larger when the ratio is larger.

[0042] Relying solely on Hamming distance to determine whether the first anomaly is a new anomaly may not be accurate or reliable in some cases. Therefore, the probability obtained in step S106 can be adjusted based on the specific circumstances of the first anomaly. Figure 1 As shown, method 100 further includes: at step S108, modifying the probability obtained in step S106 based on at least one of the length of the text of the content of the first anomaly and the instability of the first component.

[0043] On one hand, the text length of anomalous content can affect Hamming distance-based judgments. When the text of anomalous content being compared pairwise is short, a difference of just one character can lead to a large Hamming distance, thus making it more likely to be judged as a new anomalous content. Therefore, in order to suppress the influence of the text length of anomalous content on the bias of Hamming distance-based judgments, the probability obtained in step S106 can be adjusted based on the text length of the anomalous content.

[0044] In some embodiments, the length of the text of the first abnormal content can be mapped to the interval [0,1], and the probability can be adjusted based on the mapping result. Any suitable mapping can be used, as long as the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases.

[0045] In some embodiments, the difference between the length of the text of the first abnormal content and a preset text length threshold can be mapped to the [0,1] interval, and the probability can be adjusted based on the mapping result. Any suitable mapping can be used, as long as it is configured such that the mapping result becomes smaller as the difference decreases. The preset text length threshold can be a value determined by testing. In some examples herein, a preset text length threshold of 150 can be used for illustration. It should also be noted that when the length of the text of the first abnormal content is greater than the preset text length threshold, the difference between the length of the text of the first abnormal content and the preset text length threshold is positive; when the length of the text of the first abnormal content is less than the preset text length threshold, the difference between the length of the text of the first abnormal content and the preset text length threshold is negative.

[0046] In some embodiments, the probability obtained in step S106 for correcting the length of the text based on the content of the first anomaly may further include: when the length of the text of the content of the first anomaly is not less than a preset text length threshold, not correcting the probability based on the length of the text of the first anomaly; when the length of the text of the content of the first anomaly is less than the preset text length threshold, mapping the difference between the length of the text of the first anomaly and the preset text length threshold to the interval [0,1], and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the difference is smaller. In some examples, the mapping in the above embodiments can be implemented using the NReLU function. For example, suppose the length of the text of the first anomaly is X3, and the preset text length threshold is L. t ,but The Min function is used to find the minimum value, and the ABS function is used to find the absolute value. Therefore, the NReLU(X3) corrected probability P0 can be expressed as... Therefore, when X3≥L t When X3 is zero, NReLU(X3) = 0, P = P0, which represents the length correction probability of text not based on the first anomaly. This can also be understood as the mapping result always being 1. <L t When X3 is smaller, NReLU(X3) is larger (in the [0,1] interval), indicating an increased tendency to classify it as a new anomaly. Therefore, to overcome this bias, the probability can be adjusted based on the length of the first anomaly's text; here, the probability is adjusted to be smaller as X3 decreases. In some embodiments, a weighting parameter can also be set to adjust the effect of the length of the first anomaly's text on the probability adjustment. For example, W3 is located in the [0,1] interval and serves as a weighting parameter to adjust the influence of NReLU(X3). W3 can be set empirically based on specific circumstances, or it can be learned through machine learning models. In some examples, W3 can be set to a default value of 0.2. Of course, other functions used for mapping to the [0,1] interval are also feasible, as long as the difference between the length of the text in the first anomaly and the preset text length threshold is minimized.

[0047] In other embodiments, the ratio between the length of the text of the first abnormal content and a preset text length threshold can be mapped to the [0,1] interval, and the probability can be adjusted based on the mapping result. Any suitable mapping can be used, as long as the mapping is configured such that the mapping result becomes smaller as the ratio decreases.

[0048] In other embodiments, when the length of the text of the first abnormal content is not less than a preset text length threshold, the probability is not adjusted based on the length of the text of the first abnormal content; when the length of the text of the first abnormal content is less than the preset text length threshold, the ratio between the length of the text of the first abnormal content and the preset text length threshold is mapped to the [0,1] interval, and the probability is adjusted based on the mapping result. Any suitable mapping can be used, as long as the mapping is configured such that the mapping result becomes smaller as the ratio decreases.

[0049] In other embodiments, when the length of the text of the first abnormal content is not less than a preset text length threshold, the probability is not adjusted based on the length of the text of the first abnormal content; when the length of the text of the first abnormal content is less than the preset text length threshold, the length of the text of the first abnormal content is mapped to the [0,1] interval, and the probability is adjusted based on the mapping result. Any suitable mapping can be used, as long as the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases.

[0050] On the other hand, anomalies occurring when a component is stable are more likely to be novel anomalies. However, anomalies occurring when a component is unstable are often not novel anomalies, and in such cases, novel anomalies identified solely by Hamming distance and / or the length of the anomaly content text are likely to be false alarms. Therefore, to suppress the possibility of false alarms based on Hamming distance and / or the length of the anomaly content text, the probability obtained in step S106 can be adjusted based on the component's instability.

[0051] In some embodiments, the instability of the first component can be mapped to the [0,1] interval, and the probability can be adjusted based on the mapping result. Any suitable mapping can be used, as long as the mapping result is configured such that the mapping result increases as the instability of the first component decreases.

[0052] In some embodiments, when the instability of the first component is not greater than a preset instability threshold, the probability is not corrected based on the instability of the first component; when the instability of the first component is greater than the preset instability threshold, the instability of the first component is mapped to the [0,1] interval, and the probability is corrected based on the mapping result. Any suitable mapping can be used, as long as the mapping is configured such that the mapping result increases as the instability of the first component decreases. The preset instability threshold can be a value set empirically, and may have different suitable values ​​in different application scenarios. For example, the preset instability threshold can be selected as no more than 30%, or no more than 20%, or no more than 10%, or no more than 5%, or even 0%.

[0053] In some embodiments, a ticket box design can be used to calculate the instability of the first component. Specifically, a ticket box with a maximum number of tickets of N can be allocated to the first component and initialized by filling the ticket box with tickets. The remaining number of tickets in the ticket box is set to X2. If the ticket box is not full, δ tickets are added to the ticket box every preset time interval, and (δ+(N-X2)) tickets are removed from the ticket box whenever an anomaly occurs in the first component. Thus, the instability of the first component can be expressed as: For example, assuming the maximum number of tickets in the ballot box is 100, and 10 tickets are added to the box every hour when it is not full, and (10 + (100 - x2)) tickets are removed from the box whenever the first component malfunctions, then the instability of the first component can be expressed as: It lies in the interval [0,1]. Therefore, based on... The corrected probability P0 can be expressed as Or, in combination with corrections based on the length of the abnormal content text. X2 corresponds to the instability of the first component. For example, if no anomalies have occurred, X2 = N, and the instability of the first component is 0%. In this case, the probability of false alarms from P0 or P1 is low, so it is reasonable to believe that P = P0 or... As anomalies occur repeatedly within a certain timeframe, the instability of the component increases at an increasingly faster rate, and the rate at which X2 decreases also increases. In this situation, the likelihood of false alarms for P0 or P1 also increases. Therefore, to overcome such false alarms, the probability can be corrected based on the instability of the first component; here, the probability is corrected to be smaller as X2 decreases. In some embodiments, a weighting parameter can also be set to adjust the effect of the length of the first anomaly's text on the probability correction.

[0054] For example or W2 is located in the [0,1] interval and serves as a weight adjustment parameter to regulate the impact of the instability of the first component. W2 can be set empirically based on specific circumstances, or it can be learned through machine learning models. In some examples, W2 can be set to a default value of 0.5. Of course, other functions used for mapping to the [0,1] interval are also feasible, as long as the mapping result is increased when the instability of the first component is small.

[0055] Similarly, the probability can be adjusted based on the stability of the first component. For example, in the ticket box design described above, the stability of the first component can be expressed as... Then, we can also get or Among them W ′ 2 is located in the [0,1] interval and is used as a weight adjustment parameter to adjust the stability of the first component.

[0056] In some embodiments, the corrected probability P obtained in step S108 can be expressed as:

[0057]

[0058] Where X1 is the minimum Hamming distance, D t X1 is the preset Hamming distance threshold, X2 is the remaining number of votes in the first component's ballot box, N is the maximum number of votes in the first component's ballot box, X3 is the length of the text of the first abnormal content, and L is the maximum number of votes in the first component's ballot box. t W1 is a preset text length threshold, and W2 and W3 are weight adjustment parameters in the interval [0,1]. For example, in a specific non-limiting example, the modified probability P obtained in step S108 can be expressed as:

[0059] Continue to refer to Figure 1Method 100 further includes: at step S110, identifying whether the first anomaly is a new anomaly based on the modified probability obtained at step S108. In some embodiments, method 100 may further include: in response to identifying the first anomaly as a new anomaly, adding a record of the first anomaly to an anomaly list table.

[0060] In some embodiments, method 100 may further include: when the corrected probability is less than a preset probability threshold, identifying the first anomaly as a second anomaly in an anomaly list that has the minimum Hamming distance to the first anomaly; when the corrected probability is not less than the preset probability threshold, identifying the first anomaly as a new anomaly and adding a record of the first anomaly to the anomaly list, the record including descriptive attributes of the first anomaly (descriptive attributes may include, for example, the type and content of the anomaly). For example, the preset probability threshold may be a value set based on experience, and may have different suitable values ​​in different application scenarios. In some examples herein, the preset probability threshold may be selected as not less than 50%, or not less than 60%, or not less than 70%, or not less than 80%, or not less than 90%.

[0061] In some embodiments, method 100 may further include: when the corrected probability is less than a preset probability threshold, adding a record of a first exception to an exception table, the record including the primary key of the second exception in the exception list table and the transient attributes of the first exception (the transient attributes may include, for example, the location and time of the exception occurrence); when the corrected probability is not less than the preset probability threshold, adding a record of the first exception to the exception event table, the record including the primary key of the first exception in the exception list table and the transient attributes of the first exception. The exception event table can be used to record every exception that has occurred in the distributed architecture. Figure 2 The table dwd_event_error_list shown is a non-restricted example of an exception event table. Figure 2As shown, the exception event table (dwd_event_error_list) records multiple data entries for each exception event, including a primary key (or exception event primary key), exception list primary key, server IP, status, remarks, creation time, minimum Hamming distance, whether it is a new exception, and the probability of a new exception. When the corrected probability is less than a preset probability threshold, a record for the first exception can be added to the exception event table (dwd_event_error_list), and the primary key of the second exception in the exception list table can be filled in the corresponding "Exception List Primary Key" field. When the corrected probability is not less than the preset probability threshold, not only can the instantaneous attributes of the first exception be recorded in the exception event table (dwd_event_error_list), but the descriptive attributes of the first exception as a new exception can also be recorded in the exception list table (dim_meta_error_list). An alarm indicating the first exception as a new exception can also be issued.

[0062] This disclosure achieves fast, accurate, and reliable anomaly identification by maintaining an anomaly list and determining the probability that a real-time anomaly is a new anomaly not recorded in the list based on Hamming distance. Furthermore, it modifies the probability determined based on Hamming distance by using at least one of the anomaly content text length and the instability of the component containing the anomaly. Especially when applied to large-scale distributed architectures, it can efficiently identify various types of anomalies from different components within the distributed architecture, facilitating early detection and resolution of problems and eliminating potential system risks.

[0063] This disclosure also provides, in another aspect, an anomaly detection device for distributed architectures. For example... Figure 3 As shown, the anomaly identification device 300 includes a processing module 302, a correction module 304, and an identification module 306. The processing module 302 is configured to: convert the text of the content of a first anomaly collected in a first component of the distributed architecture into a SimHash value; determine the minimum Hamming distance among the Hamming distances between the SimHash value of the first anomaly and the SimHash values ​​of each anomaly in the anomaly list table, the anomaly list table being used to maintain non-repeating anomalies that have occurred in the distributed architecture; and convert the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly. The correction module 304 is configured to correct the probability obtained by the processing module 302 based on at least one of the length of the text of the first anomaly content and the instability of the first component. The identification module 306 is configured to identify whether the first anomaly is a new anomaly based on the corrected probability obtained by the correction module 304.

[0064] In some embodiments, the processing module 302 is configured to map the difference or ratio between the minimum Hamming distance and a preset Hamming distance threshold to the [0,1] interval, and use the mapping result as a probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result is larger when the difference or ratio is larger.

[0065] In some embodiments, the processing module 302 is configured to map the minimum Hamming distance to the interval [0,1] and use the mapping result as a probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result increases as the minimum Hamming distance increases.

[0066] In some embodiments, the correction module 304 is configured to correct the probability obtained by the processing module 302 based on the length of the text of the first abnormal content by one of the following operations: mapping the length of the text of the first abnormal content to the interval [0,1] and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases; or mapping the difference or ratio between the length of the text of the first abnormal content and a preset text length threshold to the interval [0,1] and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the difference or ratio decreases.

[0067] In some embodiments, the correction module 304 is configured to correct the probability obtained by the processing module 302 based on the length of the text of the content of the first anomaly by:

[0068] When the length of the text of the first abnormal content is not less than a preset text length threshold, the probability is not adjusted based on the length of the text of the first abnormal content;

[0069] When the length of the text of the first abnormal content is less than the preset text length threshold:

[0070] Map the difference or ratio between the length of the first abnormal content's text and the preset text length threshold to the [0,1] interval, and adjust the probability based on the mapping result, wherein the mapping is configured such that the mapping result becomes smaller as the difference or ratio decreases; or

[0071] The length of the text of the first abnormal content is mapped to the interval [0,1], and the probability is corrected based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases.

[0072] In some embodiments, the correction module 304 is configured to correct the probability obtained by the processing module 302 based on the instability of the first component by mapping the instability of the first component to the interval [0,1] and correcting the probability based on the mapping result, wherein the mapping is configured such that the mapping result is larger as the instability of the first component decreases.

[0073] In some embodiments, the correction module 304 is configured to correct the probability obtained by the processing module 302 based on the instability of the first component by the following operations: when the instability of the first component is not greater than a preset instability threshold, the probability is not corrected based on the instability of the first component; when the instability of the first component is greater than the preset instability threshold, the instability of the first component is mapped to the [0,1] interval, and the probability is corrected based on the mapping result, wherein the mapping is configured to make the mapping result larger as the instability of the first component decreases.

[0074] In some embodiments, the correction module 304 is configured to map the instability of the first component to the interval [0,1] by: allocating a ticket box with a maximum number of tickets of N to the first component and initializing the ticket box by filling it with tickets, wherein the remaining number of tickets in the ticket box is set to X2; adding δ tickets to the ticket box every preset time period when the ticket box is not full; and removing (δ+(N-X2)) tickets from the ticket box whenever the first component experiences an anomaly; and wherein the instability of the first component is represented as...

[0075] In some embodiments, the corrected probability P obtained by the correction module 304 is expressed as

[0076] Where X1 is the minimum Hamming distance, D t X1 is the preset Hamming distance threshold, X2 is the remaining number of votes in the first component's ballot box, N is the maximum number of votes in the first component's ballot box, X3 is the length of the text of the first abnormal content, and L is the maximum number of votes in the first component's ballot box. t W1 is the preset text length threshold, and W2 and W3 are weight adjustment parameters in the range [0,1].

[0077] In some embodiments, the identification module 306 is further configured to: add a record of the first exception to the exception list table in response to identifying the first exception as a new exception.

[0078] In some embodiments, the identification module 306 is further configured to: when the corrected probability obtained by the correction module 304 is less than a preset probability threshold, identify the first anomaly as a second anomaly in the anomaly list table that has the minimum Hamming distance to the first anomaly; when the corrected probability obtained by the correction module 304 is not less than the preset probability threshold, identify the first anomaly as a new anomaly and add a record of the first anomaly to the anomaly list table, the record including the descriptive attributes of the first anomaly.

[0079] In some embodiments, the identification module 306 is further configured to: when the corrected probability obtained by the correction module 304 is less than a preset probability threshold, add a record of a first exception to the exception event table, the record including the primary key of the second exception in the exception list table and the instantaneous attribute of the first exception, the exception event table being used to record each exception that has occurred in the distributed architecture; when the corrected probability obtained by the correction module 304 is not less than the preset probability threshold, add a record of a first exception to the exception event table, the record including the primary key of the first exception in the exception list table and the instantaneous attribute of the first exception.

[0080] The embodiment of the anomaly identification device 300 is basically similar to the embodiment of the aforementioned method 100, so it will not be described again here. For relevant details, please refer to the description in the method embodiment section.

[0081] This disclosure also provides a computing device for anomaly identification in a distributed architecture, which may include one or more processors and a memory storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the anomaly identification method for a distributed architecture according to any of the foregoing embodiments of this disclosure. Figure 4As shown, computing device 400 may include one or more processors 402 and memory 404 storing computer-executable instructions, which, when executed by one or more processors 402, cause one or more processors 402 to perform an anomaly detection method for a distributed architecture according to any of the foregoing embodiments of this disclosure. The one or more processors 402 may be, for example, a central processing unit (CPU) of computing device 400. The one or more processors 402 may be any type of general-purpose processor, or may be a processor specifically designed for anomaly detection in a distributed architecture, such as an application-specific integrated circuit (“ASIC”). Memory 404 may include various computer-readable media accessible by one or more processors 402. In various embodiments, memory 404 described herein may include volatile and non-volatile media, removable and non-removable media. For example, memory 404 may include any combination of random access memory (“RAM”), dynamic RAM (“DRAM”), static RAM (“SRAM”), read-only memory (“ROM”), flash memory, cache memory, and / or any other type of non-transitory computer-readable media. The memory 404 may store instructions that, when executed by the processor 402, cause the processor 402 to execute the anomaly identification method for a distributed architecture according to any of the foregoing embodiments of this disclosure.

[0082] This disclosure also provides a non-transient storage medium having computer-executable instructions stored thereon, which, when executed by a computer, cause the computer to perform an anomaly identification method for a distributed architecture according to any of the foregoing embodiments of this disclosure.

[0083] The terms “left,” “right,” “front,” “back,” “top,” “bottom,” “upper,” “lower,” “high,” “lower,” etc., used in the specification and claims, if present, are for descriptive purposes and not necessarily for describing unchanging relative positions. It should be understood that such terms are interchangeable where appropriate, so that embodiments of this disclosure described herein can operate, for example, in orientations different from those shown or otherwise described herein. For example, when the device in the drawings is reversed, a feature previously described as “above” other features may now be described as “below” other features. The device may also be oriented in other ways (rotated 90 degrees or in other orientations), in which case the relative spatial relationships will be interpreted accordingly.

[0084] In the specification and claims, when an element is described as being "on top of," "attached to," "connected to," "coupled to," or "in contact with" another element, the element may be directly located on top of, directly attached to, directly connected to, directly coupled to, or directly in contact with the other element, or one or more intermediate elements may be present. Conversely, when an element is described as being "directly" located on top of, directly attached to, directly connected to, directly coupled to, or directly in contact with another element, no intermediate elements are present. In the specification and claims, when a feature is arranged "adjacent" to another feature, it may mean that a feature has a portion overlapping with the adjacent feature or a portion located above or below the adjacent feature.

[0085] As used herein, the term "exemplary" means "serving as an example, instance, or illustration," and not as a "model" to be precisely copied. Any implementation described herein by example is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, this disclosure is not limited to any theory expressed or implied as given in the art, background, summary of the invention, or detailed description.

[0086] As used herein, the term "substantially" means any minor variation resulting from design or manufacturing defects, device or component tolerances, environmental influences, and / or other factors. The term "substantially" also allows for differences from the perfect or ideal situation due to parasitic effects, noise, and other practical considerations that may exist in the actual implementation.

[0087] Additionally, terms such as “first,” “second,” etc., may be used in this document for reference purposes only and are not intended to be limiting. For example, unless the context clearly indicates otherwise, the words “first,” “second,” and other such numerical terms relating to structures or elements do not imply order or sequence.

[0088] It should also be understood that when the term “including / contains” is used herein, it indicates the presence of the indicated feature, whole, step, operation, unit and / or component, but does not preclude the presence or addition of one or more other features, wholes, steps, operations, units and / or components and / or combinations thereof.

[0089] In this disclosure, the term “provide” is used broadly to cover all ways of obtaining an object, and therefore “provide an object” includes, but is not limited to, “purchasing,” “preparing / manufacturing,” “arranging / setting up,” “installing / assembling,” and / or “ordering” an object.

[0090] As used herein, the term “and / or” includes any and all combinations of one or more of the listed items in association. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise.

[0091] Those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial overlap in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and substitutions are equally possible. Aspects and elements of all the embodiments disclosed above may be combined in any way and / or in combination with aspects or elements of other embodiments to provide multiple additional embodiments. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0092] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. The various embodiments disclosed herein can be combined in any way without departing from the spirit and scope of this disclosure. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. An anomaly detection method for distributed architectures, comprising: The text of the first anomaly collected in the first component of the distributed architecture is converted into a SimHash value; Determine the minimum Hamming distance among the Hamming distances between the SimHash value of the first exception and the SimHash value of each exception in the exception list table, wherein the exception list table is used to maintain non-repeating exceptions that have occurred in the distributed architecture. The minimum Hamming distance is converted into a probability indicating that the first anomaly is a new anomaly, wherein the larger the minimum Hamming distance, the greater the probability that the first anomaly is a new anomaly; The probability is corrected based on the length of the text containing the first anomaly and the instability of the first component; and The first anomaly is identified as a new anomaly based on the modified probability.

2. The anomaly identification method according to claim 1 further includes: In response to recognizing the first exception as a new exception, a record for the first exception is added to the exception list table.

3. The anomaly identification method according to claim 1 further includes: When the corrected probability is less than a preset probability threshold, the first anomaly is identified as a second anomaly in the anomaly list that has the minimum Hamming distance from the first anomaly. When the corrected probability is not less than the preset probability threshold, the first anomaly is identified as a new anomaly and a record of the first anomaly is added to the anomaly list table. The record includes the descriptive attributes of the first anomaly.

4. The anomaly identification method according to claim 3 further includes: When the corrected probability is less than the preset probability threshold, a record of the first exception is added to the exception event table. The record includes the primary key of the second exception in the exception list table and the instantaneous attribute of the first exception. The exception event table is used to record every exception that has occurred in the distributed architecture. When the corrected probability is not less than the preset probability threshold, a record of the first anomaly is added to the anomaly event table. The record includes the primary key of the first anomaly in the anomaly list table and the instantaneous attributes of the first anomaly.

5. The anomaly identification method according to any one of claims 1 to 4, wherein, Converting the minimum Hamming distance into a probability indicating that the first anomaly is a new anomaly includes one of the following: Map the minimum Hamming distance to the interval [0, 1], and use the mapping result as the probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result increases as the minimum Hamming distance increases; or The difference or ratio between the minimum Hamming distance and a preset Hamming distance threshold is mapped to the interval [0, 1], and the mapping result is used as the probability indicating that the first anomaly is a new anomaly, wherein the mapping is configured such that the mapping result is larger when the difference or ratio is larger.

6. The anomaly identification method according to any one of claims 1 to 4, wherein, The probability is adjusted based on the length of the text containing the first anomaly, including one of the following: Map the length of the text of the first abnormal content to the interval [0, 1], and adjust the probability based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases; or The difference or ratio between the length of the text of the first abnormal content and a preset text length threshold is mapped to the interval [0,1], and the probability is corrected based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the difference or ratio is smaller.

7. The anomaly identification method according to any one of claims 1 to 4, wherein, The probability is adjusted based on the length of the text containing the first anomaly, including: When the length of the text of the first abnormal content is not less than a preset text length threshold, the probability is not adjusted based on the length of the text of the first abnormal content; When the length of the text of the first abnormal content is less than the preset text length threshold: Map the difference or ratio between the length of the first abnormal content's text and the preset text length threshold to the interval [0, 1], and adjust the probability based on the mapping result, wherein the mapping is configured such that the mapping result decreases as the difference or ratio decreases; or The length of the text of the first abnormal content is mapped to the interval [0, 1], and the probability is corrected based on the mapping result, wherein the mapping is configured such that the mapping result is smaller as the length of the text of the first abnormal content decreases.

8. The anomaly identification method according to any one of claims 1 to 4, wherein, Correcting the probability based on the instability of the first component includes: The instability of the first component is mapped to the interval [0, 1], and the probability is corrected based on the mapping result, wherein the mapping is configured such that the mapping result is larger as the instability of the first component decreases.

9. The anomaly identification method according to any one of claims 1 to 4, wherein, Correcting the probability based on the instability of the first component includes: When the instability of the first component is not greater than a preset instability threshold, the probability is not corrected based on the instability of the first component; When the instability of the first component is greater than the preset instability threshold, the instability of the first component is mapped to the interval [0, 1], and the probability is corrected based on the mapping result, wherein the mapping is configured to make the mapping result larger as the instability of the first component decreases.

10. The anomaly identification method according to claim 8, wherein, Mapping the instability of the first component to the [0,1] interval includes: The maximum number of votes that can be allocated to the first component is The ticket box is initialized by filling it with tickets, wherein the remaining number of tickets in the ticket box is set to... If the ballot box is not full, ballots will be added to the ballot box at preset time intervals. Tickets are collected and removed from the ticket box whenever the first component malfunctions. Tickets, and wherein the instability of the first component is represented as .

11. The anomaly identification method according to claim 10, wherein, The modified probability P is expressed as: in, It is the minimum Hamming distance, It is a preset Hamming distance threshold. This is the number of remaining tickets in the ticket box of the first component. This is the maximum number of tickets that can be collected from the ticket box in the first component. It is the length of the text containing the content of the first anomaly. It is a preset text length threshold, and and It is a weight adjustment parameter in the range [0, 1].

12. An anomaly detection device for a distributed architecture, comprising: The processing module is configured as follows: The text of the first anomaly collected in the first component of the distributed architecture is converted into a SimHash value; Determine the minimum Hamming distance among the Hamming distances between the SimHash value of the first exception and the SimHash value of each exception in the exception list table, wherein the exception list table is used to maintain non-repeating exceptions that have occurred in the distributed architecture. The minimum Hamming distance is converted into a probability indicating that the first anomaly is a new anomaly, wherein the larger the minimum Hamming distance, the greater the probability that the first anomaly is a new anomaly; The correction module is configured to correct the probability based on the length of the text of the content of the first anomaly and the instability of the first component; and The identification module is configured to identify whether the first anomaly is a new anomaly based on the modified probability.

13. A computing device for anomaly identification in a distributed architecture, comprising: One or more processors; as well as A memory storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform an anomaly detection method for a distributed architecture according to any one of claims 1 to 11.

14. A non-transient storage medium having computer-executable instructions stored thereon, the computer-executable instructions causing a computer to perform an anomaly identification method for a distributed architecture according to any one of claims 1 to 11 when executed by the computer.

Images