An abnormality detection and alarm pressure sensitive method based on multi-source log correlation analysis

Abstract

This invention discloses an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis, belonging to the field of log anomaly detection technology. The method includes converting log entries of different formats into standardized log events with unified timestamps, event types, and source host identifiers; periodically extracting resource performance indicator sequences and calculating statistical baselines and dynamic thresholds; using a sliding time window mechanism to match multiple time-series log events within the window according to event correlation rules to generate composite alarm events; detecting indicator out-of-bounds events and spatiotemporally associating them with composite alarm events based on time proximity and source host identifier consistency; merging the two after successful association to generate a review anomaly event. This method can capture the inherent correlation of log events, filter invalid alarms, reduce the probability of false positives, and improve the efficiency and reliability of anomaly investigation.

Description

Technical Field

[0001] This invention belongs to the field of log anomaly detection technology, specifically an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis. Background Technology

[0002] In log anomaly detection and alerting scenarios, existing technologies typically standardize logs from different sources and in different formats, converting log entries into log events of a uniform format. Resource performance metrics are then extracted from the standardized logs, and by setting fixed thresholds or simple statistical baselines, out-of-bounds metric detection is performed to generate alert events. Simultaneously, some technologies independently monitor single types of log events, directly generating independent alerts when abnormal log events or out-of-bounds metric detections are detected, prompting relevant personnel to investigate the anomaly.

[0003] In existing technical solutions, log event processing is mostly done through isolated monitoring, failing to correlate and match multiple log events that occur simultaneously or sequentially along a time-series dimension. This results in mostly single-event alarms that fail to reflect the correlation between events, easily generating a large number of unrelated alarms and increasing the burden of troubleshooting. Furthermore, the matching of metric out-of-bounds events with log alarm events lacks clear correlation criteria. Relying solely on single-dimensional anomaly judgment cannot distinguish between valid anomalies and invalid alarms, leading to low confidence in anomaly detection, difficulty in accurately locating true system anomalies, and an inability to meet the accuracy and efficiency requirements of complex systems for anomaly detection. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art;

[0005] To address this, the present invention proposes an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis, comprising:

[0006] Convert log entries of different formats into standardized log events with a unified timestamp, unified event type, and unified source host identifier;

[0007] From the standardized log events, resource performance index sequences are periodically extracted according to predefined resource performance index extraction rules, and the baseline value of the statistical baseline and the upper and lower limits of the dynamic threshold are calculated for each resource performance index sequence.

[0008] Using a sliding time window mechanism, the standardized log events are processed in a time-series manner. Within a window, multiple standardized log events that occur simultaneously or successively are matched according to pre-configured event association rules. When the combination conditions of the event association rules are met, a composite alarm event is generated.

[0009] From the resource performance index sequence, index values ​​that exceed the statistical baseline and dynamic threshold are detected, index out-of-bounds events are generated, and each index out-of-bounds event is spatiotemporally correlated with the composite alarm event. The spatiotemporal correlation is matched based on the temporal proximity of the index out-of-bounds event and the consistency of the source host identifier between the composite alarm event and the composite alarm event.

[0010] After the spatiotemporal correlation is successfully established, the matched index out-of-bounds event is merged with the composite alarm event to generate an abnormal event to be reviewed.

[0011] Furthermore, from the standardized log events, resource performance indicator sequences are periodically extracted according to predefined resource performance indicator extraction rules, and a baseline value for statistical baselines and upper and lower limits for dynamic thresholds are calculated for each resource performance indicator sequence, including:

[0012] For each type of log, the corresponding regular expression pattern is applied to capture numerical resource usage data from the text content of the standardized log events;

[0013] All captured resource usage data are organized into time series data points according to their source host identifier, resource indicator type, and timestamp to form the resource performance indicator sequence.

[0014] For each of the resource performance index sequences, the quantile statistical method is used to calculate its multiple quantile values ​​in the past full natural day, and the 50th percentile is used as the benchmark value of the statistical baseline.

[0015] Based on the baseline value of the statistical baseline and the standard deviation of the fluctuation of the resource performance index sequence in the past statistical period, the upper limit and lower limit of the dynamic threshold are calculated. The upper limit of the dynamic threshold is the baseline value plus a certain number of standard deviations, and the lower limit of the dynamic threshold is the baseline value minus a certain number of standard deviations.

[0016] Furthermore, the sliding time window mechanism is used to perform time-series processing on the standardized log events. Within a window, multiple standardized log events that occur simultaneously or successively are matched according to pre-configured event association rules. When the combination conditions of the event association rules are met, a composite alarm event is generated, including:

[0017] Set the window length of the sliding time window, whereby the window length is defined as a fixed duration tracing back from the timestamp of the currently processed standardized log event;

[0018] Each standardized log event is read sequentially from the streaming log queue and placed into a sliding time window based on the event occurrence time;

[0019] Query the pre-configured event association rules, which define combination patterns for various event types, the maximum allowed time interval for each event within the combination, and specific attribute conditions that each event must meet;

[0020] Iterate through all the standardized log events within the current sliding time window to find if there exists an event set whose event type, event attributes, and time difference between events completely match the definition of a certain event association rule.

[0021] When a set of events that perfectly matches is found, the unique identifiers, event types, and core attributes of all events in the set are aggregated to create a new log entity as the composite alarm event. The composite alarm event contains summary information and association confidence of multiple associated standardized log events, and is assigned an alarm type label that reflects the abnormal scenario represented by the event association rule.

[0022] Furthermore, from the resource performance indicator sequence, indicator values ​​exceeding the statistical baseline and dynamic threshold are detected, and indicator out-of-bounds events are generated, including:

[0023] Receive data points of the newly collected resource performance index sequence in real time;

[0024] For each new data point, based on its source host identifier and resource indicator type, the corresponding baseline value of the statistical baseline, the upper limit value and the lower limit value of the dynamic threshold are queried from the stored data;

[0025] The values ​​of the newly arrived data points are compared with the upper and lower limits of the queried dynamic threshold, respectively;

[0026] When the value of the newly arrived data point is greater than the upper limit of the dynamic threshold or less than the lower limit of the dynamic threshold, the corresponding data point is determined to have experienced an out-of-bounds event.

[0027] Record the resource metric type, source host identifier, out-of-bounds value, out-of-bounds direction, timestamp of the out-of-bounds occurrence, and the dynamic threshold exceeded corresponding to the out-of-bounds occurrence. Encapsulate all the recorded information into a single out-of-bounds event and trigger the subsequent spatiotemporal correlation process.

[0028] Furthermore, each of the aforementioned metric out-of-bounds events is spatiotemporally correlated with the composite alarm event. This spatiotemporal correlation is based on matching the temporal proximity and source host identifier consistency between the metric out-of-bounds event and the composite alarm event, including:

[0029] Whenever an out-of-bounds event of the aforementioned indicator is generated, a time-related interval is immediately defined centered on its timestamp;

[0030] Retrieve all composite alarm events whose timestamps fall within the time association interval from the event cache, the event cache being used to temporarily store recently generated composite alarm events;

[0031] From the retrieved composite alarm events, further filter out events whose source host identifier is exactly the same as the source host identifier of the index out-of-bounds event;

[0032] Calculate the absolute time difference between the timestamp of each selected composite alarm event and the timestamp of the index out-of-bounds event;

[0033] The absolute time difference is compared with the preset maximum allowable association time difference. If the absolute time difference is less than or equal to the maximum allowable association time difference, it is determined that the composite alarm event and the indicator out-of-bounds event are successfully spatiotemporally associated, and the association relationship between the composite alarm event and the indicator out-of-bounds event is established.

[0034] Furthermore, after the spatiotemporal correlation is successfully established, the matched indicator out-of-bounds event is merged with the composite alarm event to generate an anomaly event to be reviewed, including:

[0035] Extract the detailed information of the indicator out-of-bounds event, including the type of resource indicator that went out of bounds, the value of the out-of-bounds event, and the direction of the out-of-bounds event;

[0036] Extract detailed information of the composite alarm event that has been successfully correlated in time and space, including its alarm type label and summary information of the associated standardized log event;

[0037] Create a new data structure to represent the exception event to be reviewed;

[0038] The detailed content of the indicator out-of-bounds event, the detailed content of the composite alarm event, and the time difference between the indicator out-of-bounds event and the composite alarm event are all filled into the data structure of the abnormal event to be reviewed. The abnormal event to be reviewed includes the indicator details of the indicator out-of-bounds event, the summary of the associated log event of the composite alarm event, and the relationship between the indicator out-of-bounds event and the composite alarm event.

[0039] An initial confidence score is calculated for the anomaly event to be reviewed. The confidence score is derived by combining the correlation confidence of the composite alarm event, the out-of-bounds magnitude of the indicator out-of-bounds event, and the time difference between the indicator out-of-bounds event and the composite alarm event.

[0040] Further, the upper and lower limits of the dynamic threshold are calculated based on the baseline value of the statistical baseline and the standard deviation of the resource performance index sequence over the past statistical period, including:

[0041] The coefficient factor of the fluctuation standard deviation is determined based on the predefined sensitivity parameter.

[0042] From historical storage, obtain all values ​​of the resource performance index sequence within the most recent complete statistical period, and calculate the standard deviation of the corresponding set of values ​​as the fluctuation standard deviation;

[0043] The upper limit of the dynamic threshold is obtained by adding the product of the baseline value of the statistical baseline and the standard deviation of the fluctuation multiplied by the coefficient factor.

[0044] The lower limit of the dynamic threshold is obtained by subtracting the product of the baseline value of the statistical baseline and the product of the standard deviation of fluctuation and the coefficient factor.

[0045] Furthermore, the absolute time difference between the timestamp of each composite alarm event selected through calculation and the timestamp of the indicator out-of-bounds event includes:

[0046] Read the timestamp recorded in the index out-of-bounds event and record it as the first timestamp;

[0047] Read the timestamp recorded in one of the filtered composite alarm events and record it as the second timestamp;

[0048] Calculate the difference between the second timestamp and the first timestamp;

[0049] The absolute time difference is obtained by taking the absolute value of the calculated difference.

[0050] The calculated absolute time difference is compared with a preset maximum allowable associated time difference, which is dynamically configured according to the resource indicator type of the indicator out-of-bounds event and the alarm type label of the composite alarm event.

[0051] Furthermore, a new data structure is created as the exception event to be reviewed, including:

[0052] Define a structure that encapsulates all information about the exception event to be reviewed;

[0053] A first field is set in the structure to store the globally unique identifier of the abnormal event to be reviewed. The globally unique identifier is generated by combining the timestamp, the source host identifier, and the event type code.

[0054] A second field is set in the structure to store the detailed content of the indicator out-of-bounds event. The detailed content specifically includes the type of resource indicator that went out of bounds, the out-of-bounds value, the out-of-bounds direction, the dynamic threshold that was exceeded, and the timestamp of the indicator out-of-bounds event.

[0055] A third field is set in the structure to store the detailed content of the composite alarm event. The detailed content specifically includes the alarm type label of the composite alarm event, the list of unique identifiers of the associated standardized log events, the core attribute summary of the standardized log events, and the generation timestamp of the composite alarm event.

[0056] A fourth field is set in the structure to store the association information, which includes the absolute time difference between the indicator out-of-bounds event and the composite alarm event, and the timestamp of the association establishment.

[0057] A fifth field is set in the structure to store the initial confidence score;

[0058] Based on the definition of the structure, a new exception event object to be reviewed is instantiated in memory, and the first, second, third, fourth and fifth fields are filled with the corresponding data to complete the creation of the exception event to be reviewed.

[0059] Furthermore, the method also includes a step of applying alarm sensitivity processing to the abnormal events to be reviewed:

[0060] Establish a global alarm event queue to store all generated abnormal events pending review;

[0061] Configure a capacity threshold and a time aggregation window for the global alarm event queue;

[0062] Monitor the length of the global alarm event queue. When the number of consecutive abnormal events to be reviewed entering the queue within the time aggregation window causes the queue length to exceed the capacity threshold, initiate pressure sensitivity analysis.

[0063] The pressure sensitivity analysis clusters all newly entered anomaly events to be reviewed within the time aggregation window according to their source host identifier and anomaly type.

[0064] For all abnormal events to be reviewed within the same group, their common features are extracted and merged to generate an aggregated alarm event. The aggregated alarm event includes the total number of events in the same group, the first occurrence time, the last occurrence time, and a common description of the resource indicators or log events involved.

[0065] Remove the individual aggregated exception events to be reviewed from the global alarm event queue, insert the generated aggregated alarm event into the queue, and then extract the events from the head of the queue in the order of their occurrence for subsequent alarm notifications.

[0066] The pressure sensitivity analysis clusters all newly entered anomaly events to be reviewed within the time aggregation window according to their source host identifier and anomaly type, including:

[0067] Read each pending exception event that enters within the time aggregation window;

[0068] Parse the source host identifier contained in each anomaly event to be reviewed, and the alarm type label extracted from the composite alarm event or the resource indicator type extracted from the indicator out-of-bounds event;

[0069] Establish a hash mapping table with the source host identifier as the primary key and the exception type as the subkey;

[0070] Each anomaly event to be reviewed is categorized into the corresponding entry in the hash mapping table based on its source host identifier and anomaly type. Events under the same entry constitute an initial cluster group.

[0071] Iterate through all initial cluster groups. For groups whose number of events reaches the preset minimum aggregation threshold, identify them as valid cluster groups for subsequent merging to generate aggregated alarm events.

[0072] Compared with the prior art, the beneficial effects of the present invention are:

[0073] A sliding time window mechanism is employed to process standardized log events in a time-series manner. Within a single window, multiple standardized log events occurring simultaneously or sequentially are matched according to pre-configured event association rules. When the combined conditions of the event association rules are met, a composite alarm event is generated. By matching multiple events within a time-series window, the limitations of isolated monitoring of single events are overcome. This approach captures the inherent relationships between different log events, reduces the generation of unrelated single alarms, and enables alarm information to more accurately reflect the overall state of system anomalies. It also avoids the redundancy in troubleshooting caused by isolated alarms, thereby improving the efficiency of anomaly investigation.

[0074] This method detects resource performance indicator values ​​exceeding statistical baselines and dynamic thresholds from a sequence of indicators, generating indicator out-of-bounds events. Each indicator out-of-bounds event is then spatiotemporally correlated with a composite alarm event. The correlation process matches the two events based on their temporal proximity and the consistency of their source host identifiers. Upon successful correlation, the two events are merged to generate an anomaly event awaiting review. Through this dual-dimensional spatiotemporal correlation of time and host identifiers, precise matching of indicator anomalies with log event anomalies is achieved. Indicator out-of-bounds events unrelated to composite alarms are filtered out, invalid anomaly alerts are eliminated, and the generated anomaly events accurately correspond to actual system anomalies, reducing the probability of false positives and making the anomaly detection results more reliable and providing more precise guidance for anomaly troubleshooting. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating the steps of an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis as described in this invention.

[0076] Figure 2 A flowchart for extracting indicator sequences and calculating thresholds;

[0077] Figure 3 A flowchart for generating metric out-of-bounds events. Detailed Implementation

[0078] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] See Figure 1 This invention provides an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis, the specific method including:

[0080] The system parses and cleans raw log entries from various sources, including operating systems, applications, and network devices, extracting key fields and converting all log entries into standardized log events with a unified timestamp format, unified event type encoding, and unified source host identifier. From these standardized log events, based on predefined resource performance indicator extraction rules for different log types, the system periodically scans and captures numerical sequences reflecting system resource usage, forming resource performance indicator sequences. For each resource performance indicator sequence, a statistical baseline and dynamic threshold are calculated. A sliding time window mechanism is used to process the standardized log event stream in a time sequence. Within a time window, based on pre-configured event association rules, pattern matching is performed on multiple standardized log events that occur simultaneously or sequentially within the window. When a combination of events within the window satisfies all conditions of a certain event association rule, a composite alarm event is generated. Simultaneously, from the continuously updated resource performance indicator sequences, data points exceeding their statistical baseline and dynamic threshold are detected in real time, generating indicator out-of-bounds events. Each generated metric out-of-bounds event is spatiotemporally correlated with the already generated composite alarm event. The matching criteria are the temporal proximity of the two events and whether the source host identifiers involved are consistent. When the spatiotemporal correlation match is successful, the corresponding metric out-of-bounds event and composite alarm event are merged to form an anomaly event to be reviewed that contains richer contextual information.

[0081] In one embodiment of the present invention, the text content of the standardized log event includes numerical information characterizing resource usage. See also... Figure 2For each specific log type, a corresponding regular expression pattern is applied to capture numerical resource usage data, such as CPU utilization, memory usage, and disk I / O throughput, from the text description fields of standardized log events. All captured resource usage data is organized into a series of time-sorted data points based on its source host identifier, the type of resource metric it represents, and its timestamp, forming a structured sequence of resource performance metrics. For each resource performance metric sequence, quantile statistics are used to calculate multiple quantile values ​​within the past full calendar day (24 hours), with the 50th percentile, or median, serving as the baseline value for the sequence's statistical calculations. Based on the established baseline value and the standard deviation of the resource performance metric sequence's numerical fluctuation over the past full statistical period, the upper and lower limits of the dynamic threshold are calculated. A coefficient factor for the standard deviation of fluctuation is determined according to predefined sensitivity parameters. All numerical samples of the resource performance metric sequence within the most recent full statistical period are retrieved from historical storage, and the standard deviation of this sample set is calculated as its standard deviation of fluctuation. The upper limit of the dynamic threshold is obtained by adding the product of the baseline value and the standard deviation of volatility multiplied by the factor. The lower limit of the dynamic threshold is obtained by subtracting the product of the baseline value and the standard deviation of volatility multiplied by the factor.

[0082] In practical implementation, an anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis involves extracting and calculating the statistical baseline and dynamic threshold of resource performance index sequences from standardized log events. Numerical information representing the system resource consumption status is embedded in the text content field of the standardized log events. For each specific type of log, such as operating system logs, middleware logs, and database logs, a matching regular expression pattern needs to be configured. The regular expression pattern is used to accurately capture numerical resource usage data from the text description of standardized log events. For example, "75" is captured from a log describing "memory usage: 75%", and "120" is captured from a log describing "disk write rate: 120MB / s". The captured resource usage data is the raw, unprocessed value. In practice, all captured resource usage data cannot exist independently; they must be standardized according to three key dimensions: the source host identifier indicates the physical or logical server from which the data originated, the resource metric type defines the metric category represented by the data, and the timestamp records the precise moment the data was generated. The standardization process encapsulates each data point into a structured object containing these three dimensions and arranges these objects in chronological order according to their timestamps, ultimately forming a sequence of resource performance metrics with clear semantics and temporal relationships. In practice, for each resource performance metric sequence, the calculation of the baseline value requires selecting a representative historical time period. A complete natural day, or 24 hours, is used as this calculation period. Quantile statistics are used to process all historical data points within this period. Quantile statistics can describe the distribution of data throughout the interval. The 50th percentile, or median, is selected as the baseline value because it reflects the typical level of the resource metric within a day and is insensitive to brief peaks or troughs. In practice, the dynamic threshold is calculated based on the baseline value of the statistical baseline and introduces a volatility measure of the resource performance index sequence. All values ​​of the resource performance index sequence within the most recent complete statistical period are read from historical storage. This statistical period may be consistent with or different from the baseline calculation period. Then, the standard deviation of the volatility of these values ​​is calculated. The standard deviation of volatility quantifies the dispersion of historical data. A predefined sensitivity parameter is used to determine a coefficient factor, which is a configurable numerical multiplier. The upper limit of the dynamic threshold is obtained by adding the product of the baseline value of the statistical baseline and the standard deviation of volatility multiplied by the coefficient factor. The lower limit of the dynamic threshold is obtained by subtracting the product of the baseline value of the statistical baseline and the standard deviation of volatility multiplied by the coefficient factor.

[0083] In some embodiments, the predefined sensitivity parameter can be a globally fixed value, such as a coefficient factor set to 3, meaning the dynamic threshold will be set within three standard deviations above and below the baseline value. The sensitivity parameter can also be configured differently based on the importance of different resource metric types. For example, for core CPU utilization, the coefficient factor is set to 2 to provide more sensitive detection; for non-critical log output rate, the coefficient factor can be set to 4 to allow for a wider fluctuation range. In some embodiments, the normalization process of the resource performance metric sequence is continuous and periodic. New normalized log events are constantly generated, and the resource performance metric sequence is dynamically updated accordingly. The baseline value of the statistical baseline and the dynamic threshold are also recalculated periodically or after a certain amount of new data has accumulated to ensure that the threshold can adapt to the long-term changing trend of system load.

[0084] Optionally, the regular expression pattern used to capture numerical values ​​needs to be carefully designed to cover possible variations in the log format. For example, for CPU utilization, the pattern might need to match both "CPU:45%" and "cpuusage=45". In practice, multiple alternative regular expression patterns might need to be configured for the same resource metric and attempted to match sequentially until a numerical value is successfully captured. Optionally, the calculation of quantile statistics does not have to be limited to the 50th percentile. Before calculating the baseline value for the statistical calculation, multiple quantiles can be calculated to understand the data distribution, such as calculating the 25th, 50th, and 75th percentiles simultaneously, but ultimately the 50th percentile is selected as the baseline value output for the statistical calculation.

[0085] It is understandable that the calculation of the standard deviation of volatility directly depends on the historical data set within the selected statistical period. The length of the statistical period and the start time are configurable parameters. A common practice is to use the same 24-hour period as the baseline calculation, but the data used to calculate the standard deviation of volatility is the previous complete period immediately preceding the current moment, while the baseline calculation may use data from the previous calendar day. It is also understandable that the formula used in the dynamic threshold calculation process can be formally expressed as:

[0086]

[0087] in: This represents the upper limit of the dynamic threshold. This represents the lower limit of the dynamic threshold. The benchmark value representing the statistical baseline, This represents a coefficient factor determined by a predefined sensitivity parameter. This represents the standard deviation of the resource performance index series over the past statistical period. This formula clearly demonstrates the mathematical relationship between the dynamic threshold and the baseline, historical volatility, and the sensitivity of the configuration.

[0088] In one embodiment of the present invention, a sliding time window mechanism is used to process the standardized log event stream. The window length of the sliding time window is defined as a fixed duration, such as five or ten minutes, tracing back from the timestamp of the currently being processed standardized log event. This fixed duration is a pre-configured parameter. Standardized log events are read sequentially from a streaming log queue, which is a buffer sorted by event occurrence time. Each read standardized log event is placed into a sliding time window based on its own timestamp. The sliding time window is a logical container used to temporarily store events that occur within a specific time period for correlation analysis. The window boundary slides forward as new events arrive, and older events are moved out of the window range according to the window length. In practical implementation, the pre-configured event association rule base needs to be queried. Event association rules define various combination patterns for event types. A combination pattern describes a set of standardized log events that need to occur simultaneously or sequentially to detect a specific abnormal scenario. For example, a rule might require "Event Type A" and "Event Type B" to occur sequentially within a specific time range. The maximum allowed time interval between events within a combination pattern is also explicitly defined, limiting the maximum time difference between the occurrence times of any two events within the combination. Specific attribute conditions that each event must meet are also specified in the event association rules. These specific attribute conditions are constraints on the values ​​of fields other than the basic type in the standardized log events, such as requiring the source host to be a specific server, the event level to be "error," or the message content to contain specific keywords. In practical implementation, traversing all standardized log events within the current sliding time window is the core step of association matching. The system needs to check if an event set exists in the window. The event types within the event set must completely match all types required by the combination patterns defined in the event association rules. The specific attributes of each event within the event set must satisfy the attribute conditions defined for the corresponding event type in the event association rules. The time difference between events within the event set must be less than or equal to the maximum time interval defined in the event association rules. In practice, when an event set that fully conforms to the event association rule definition is found, the system will perform an aggregation operation to generate a composite alarm event. The unique identifiers of all events in the event set are collected and recorded in the composite alarm event. The event types and core attributes of all events in the event set are extracted, summarized, and stored in the composite alarm event. A new, independent log entity is created as the composite alarm event. The composite alarm event contains summary information of multiple associated standardized log events and an association confidence score calculated by the system. The composite alarm event is assigned an alarm type label, which directly reflects the abnormal scenario represented by the event association rule that triggered this association, such as "suspected privilege escalation attack" or "application service chain failure".

[0089] In some embodiments, the window length of the sliding time window can be configured differently according to different event association rules. For rules that need to detect rapid and continuous attack steps, the window length can be set to the second level; for rules that detect slowly spreading system faults, the window length can be set to the minute or even hour level. In some embodiments, the pre-configured event association rules can exist in the form of structured configuration files or database records. Each rule includes a unique rule identifier, rule description, a list of event combination patterns, time interval constraints, and attribute condition assertions.

[0090] Optionally, when traversing events within the current sliding time window for matching, optimized graph matching or state machine algorithms can be used to improve matching efficiency under high event traffic. For example, event association rules can be pre-compiled into state transition tables, and streaming events can drive state changes. When a state reaches its final state, it indicates a successful match. Optionally, the association confidence of composite alarm events can be calculated based on the accuracy of the matching. For example, if the event association rule requires three events but only matches two key events, the association confidence of the generated composite alarm event can be lower than the case where all events are perfectly matched.

[0091] The sliding window's logic is understandable: whenever a new standardized log event arrives, the window's start time is updated to the event's timestamp minus the window length, and the end time is the event's timestamp. All events with timestamps earlier than the new start time are removed from the current window's consideration. The maximum time interval defined in the event association rule and the sliding window's length are two independent but related concepts. The window length is the maximum time range the system uses to retrieve candidate events, while the maximum time interval is a stricter requirement for the tightness of a successfully matched set of events. The window length needs to be greater than or equal to the maximum time interval defined in the rule to ensure that related events have a chance to fall within the same window and be detected.

[0092] In one embodiment of the present invention, see [reference] Figure 3The system receives newly acquired resource performance indicator (RPI) data points in real time. For each newly arrived data point, based on its source host identifier and resource indicator type, it queries the stored threshold data for the corresponding statistical baseline value, upper limit value, and lower limit value of the dynamic threshold. The value of the newly arrived data point is compared with the queried upper and lower limits of the dynamic threshold. When the value of the newly arrived data point is greater than the upper limit value or less than the lower limit value, it is determined that the data point has exceeded the limit. The system records the resource indicator type, source host identifier, exceeding value, exceeding direction, timestamp of the exceeding occurrence, and the specific value of the exceeded dynamic threshold. All recorded information is encapsulated into an indicator exceeding the limit event, triggering subsequent spatiotemporal correlation processes.

[0093] In its implementation, an anomaly detection and alarm sensitivity method based on multi-source log correlation analysis includes the steps of detecting out-of-bounds events from a resource performance indicator sequence and generating events. A real-time receiving module continuously monitors newly acquired data points from the data acquisition pipeline. Each newly acquired data point is a data structure containing a source host identifier, resource indicator type, value, and timestamp. In practice, for each newly arrived data point, the system performs a query operation from a persistently stored threshold data warehouse based on the two key dimensions carried in the data point: the source host identifier and the resource indicator type. The goal of the query operation is to obtain the baseline value, the upper limit value, and the lower limit value of the dynamic threshold that perfectly correspond to this specific host and resource indicator. These thresholds are pre-calculated and stored based on historical data. In practice, after retrieving the corresponding threshold, the value of the newly arrived data point is compared with the upper and lower limits of the dynamic threshold, respectively. This comparison is a simple arithmetic comparison. In practice, the comparison result directly determines whether an out-of-bounds judgment is triggered. When the value of a newly arrived data point is greater than the upper limit of the dynamic threshold, or the value of a newly arrived data point is less than the lower limit of the dynamic threshold, the system determines that the corresponding data point has exceeded the boundary. The direction of the out-of-bounds judgment is determined by the comparison result: greater than the upper limit is recorded as "upward out-of-bounds," and less than the lower limit is recorded as "downward out-of-bounds." In practice, once an out-of-bounds judgment is made, the system immediately records all associated information of this event. The recorded information includes the type of resource indicator that triggered the out-of-bounds judgment, the host identifier of the data point source, the specific value at the time of the out-of-bounds judgment, whether the out-of-bounds judgment is upward or downward, the precise timestamp of the out-of-bounds judgment, and the specific value of the dynamic threshold that was exceeded. All the recorded information is encapsulated into an indicator out-of-bounds event object with a fixed format. The generation of the indicator out-of-bounds event object triggers the subsequent spatiotemporal correlation process.

[0094] In some embodiments, the threshold data warehouse can be implemented using a key-value database, where the key is a combination of the source host identifier and the resource metric type, and the value is the baseline value of the corresponding statistical baseline, the upper limit value of the dynamic threshold, and the lower limit value of the dynamic threshold. Query operations directly access the data through the composite key. In some embodiments, the real-time reception and detection process can be designed as a pipelined architecture, where each newly arrived data point is processed independently and concurrently. The processing includes querying, comparing, judging, and recording to improve system throughput. Optionally, when comparing the data point value with the upper and lower limits of the dynamic threshold, the system can be configured to include cases equal to the threshold based on the resource metric type. For example, for some metrics, a value equal to the upper limit value of the dynamic threshold is not considered out of bounds, while for other sensitive metrics, a value equal to the upper limit value of the dynamic threshold is considered out of bounds. Optionally, when recording out-of-bounds information, in addition to recording the dynamic threshold exceeded, the baseline value of the corresponding statistical baseline can also be recorded simultaneously to provide a more complete context for subsequent analysis. It can be understood that the boundary crossing judgment logic is a binary decision function. Its inputs are the data point value, the upper limit of the dynamic threshold, and the lower limit of the dynamic threshold, and the output is a Boolean value (yes / no boundary crossing). It can also be understood that the indicator boundary crossing event is an encapsulation of a single data point's abnormal state, and is one of the original materials for subsequent association with composite alarm events. Its generation frequency is directly affected by the sensitivity of the dynamic threshold and the actual fluctuation of the resource indicator. Refer to Table 1, which shows an exemplary comparison and judgment process:

[0095] Table 1: Judgment Table for Indicator Out-of-bounds Event Generation

[0096] ;

[0097] The decision logic can be formally expressed as:

[0098]

[0099] in: This indicates whether to generate a decision output for an out-of-bounds event. The value representing the newly arrived data point. This represents the upper limit of the dynamic threshold. This represents the lower limit of the dynamic threshold. When the data point value... The upper limit of the dynamic threshold Or less than the lower limit of the dynamic threshold When the decision is true, the process of generating and recording indicator out-of-bounds events is triggered.

[0100] In one embodiment of the present invention, whenever an indicator out-of-bounds event is generated, the system immediately extends a preset duration forward and backward, centered on its timestamp. The preset duration is an independently configurable parameter, such as 30 seconds, defining a time-related interval. This time-related interval is a time range with the time of the indicator out-of-bounds event as its midpoint. In a specific implementation, the system performs a retrieval operation in the event cache, which is a memory database or high-speed storage area used to temporarily store recently generated composite alarm events. The retrieval objective is to find all composite alarm events whose timestamps fall within the time-related interval defined centered on the indicator out-of-bounds event. The retrieval criteria are that the timestamp of the composite alarm event is greater than or equal to the timestamp of the indicator out-of-bounds event minus the forward extension duration, and less than or equal to the timestamp of the indicator out-of-bounds event plus the backward extension duration. In practice, the system further performs a filtering operation from the search results set. The filtering is based on the requirement that the source host identifier of the composite alarm event and the source host identifier of the current indicator out-of-bounds event must be completely matched. The filtering operation iterates through each retrieved composite alarm event, comparing its source host identifier field with that of the indicator out-of-bounds event, and only retains composite alarm events that are completely identical to proceed to the next processing stage. In practice, for each composite alarm event filtered by source host identifier, the system calculates the absolute time difference between it and the indicator out-of-bounds event. It reads the timestamp recorded in the indicator out-of-bounds event and records this timestamp as the first timestamp. It reads the timestamp recorded in a filtered composite alarm event and records this timestamp as the second timestamp. The difference between the second timestamp and the first timestamp is calculated, and the absolute value of the calculated difference is taken to obtain the absolute time difference representing the degree of proximity of the two events. In practice, the calculated absolute time difference is compared with a preset maximum allowable association time difference, which is an independent time window threshold. If the absolute time difference is less than or equal to the maximum allowable association time difference, the composite alarm event and the indicator out-of-bounds event are successfully spatiotemporally associated, and the system establishes a bidirectional association between the composite alarm event and the indicator out-of-bounds event. If the absolute time difference is greater than the maximum allowable association time difference, the pairing is ignored.

[0101] In some embodiments, the maximum permissible correlation time difference can be dynamically configured based on the resource metric type of the metric out-of-bounds event and the alarm type label of the composite alarm event. Dynamic configuration is achieved by querying a predefined mapping table, which defines the maximum permissible correlation time difference corresponding to different combinations of resource metric types and alarm type labels. For example, for a combination of resource metric type "network connections" and alarm type label "port scan," the maximum permissible correlation time difference can be configured to 2 seconds; for a combination of resource metric type "disk utilization" and alarm type label "file system error," the maximum permissible correlation time difference can be configured to 60 seconds. In some embodiments, the preset durations for forward and backward expansion of the time correlation interval can be set to asymmetrical values. For example, considering that some composite alarm events may occur as root causes before the metric out-of-bounds event, the forward expansion duration can be set to be greater than the backward expansion duration.

[0102] Optionally, a time-to-live (TTL) can be set for storing composite alarm events in the event cache. Composite alarm events exceeding the TTL will be automatically cleaned up to ensure retrieval efficiency and avoid invalid association with outdated events. Optionally, before calculating the absolute time difference, the first and second timestamps can be uniformly converted to a timestamp format with the same precision, such as uniformly converting to Unix timestamps in milliseconds, to ensure the accuracy of time calculation.

[0103] It is understandable that the calculation of the absolute time difference reflects the proximity in occurrence time between the indicator out-of-bounds event and the composite alarm event, and is an important time dimension indicator for measuring whether the two may belong to the same abnormal event chain. It is also understandable that the maximum permissible correlation time difference and the preset duration of the time correlation interval work together. The preset duration of the time correlation interval determines the initial range for retrieving candidate composite alarm events from the event cache, acting as a relatively wide filter, while the maximum permissible correlation time difference is a stricter condition for precisely filtering candidate events based on this. See Table 2, which illustrates a spatiotemporal correlation matching scenario:

[0104] Table 2: Spatiotemporal Correlation Matching Process of Indicator Out-of-bounds Events and Composite Alarm Events

[0105] ;

[0106] The calculation of absolute time difference can be formally expressed as:

[0107]

[0108] in: This represents the calculated absolute time difference. The timestamp representing the composite alarm event record is the second timestamp. The timestamp representing the event that exceeded the limit is the first timestamp. Absolute time difference. This is the input value used for subsequent comparison with the maximum allowable associated time difference in dynamic configuration.

[0109] In one embodiment of the present invention, after the spatiotemporal correlation of an indicator out-of-bounds event and a composite alarm event is successfully established, a merging and subsequent processing step is performed. Detailed content is extracted from the indicator out-of-bounds event that has completed spatiotemporal correlation. This detailed content includes the type of resource indicator that has exceeded the bounds, the specific out-of-bounds value, and whether the out-of-bounds direction is upward or downward. Detailed content is extracted from the composite alarm event that has completed spatiotemporal correlation. This detailed content includes its alarm type label and summary information of the associated standardized log event. A new data structure is created as the anomaly event to be reviewed. A structure is defined to encapsulate all information of the anomaly event to be reviewed. A first field is set in the structure to store a globally unique identifier for the anomaly event to be reviewed. This globally unique identifier is generated by combining the timestamp when the anomaly event was generated, the source host identifier, and an event type code representing the event type. A second field is set in the structure to store the detailed content of the indicator out-of-bounds event. This detailed content includes the type of resource indicator that has exceeded the bounds, the out-of-bounds value, the out-of-bounds direction, the specific value of the dynamic threshold exceeded, and the timestamp of the occurrence of the indicator out-of-bounds event. A third field is set in the structure to store the detailed content of the composite alarm event. The structure contains detailed information including the alarm type label of the composite alarm event, a list of unique identifiers for the associated standardized log events, a summary of the core attributes of the standardized log events, and the generation timestamp of the composite alarm event. A fourth field is set in the structure to store association information, including the absolute time difference between the metric out-of-bounds event and the composite alarm event, and the timestamp of the association establishment. A fifth field is set in the structure to store an initial confidence score. Based on the structure definition, a new object of the anomaly to be reviewed is instantiated in memory, and the first, second, third, fourth, and fifth fields are filled with the corresponding data, completing the creation of the anomaly to be reviewed. In specific implementation, an initial confidence score is calculated for the anomaly to be reviewed. The confidence score is calculated based on the association confidence of the composite alarm event, the out-of-bounds magnitude of the metric out-of-bounds event, and the absolute time difference between the composite alarm event and the metric out-of-bounds event. Establish a global alarm event queue to store all generated anomaly events awaiting review. Configure a capacity threshold and a time aggregation window for the global alarm event queue. The capacity threshold defines the maximum number of events the queue can accommodate before triggering stress sensitivity analysis. The time aggregation window defines a continuous observation time range to monitor the length of the global alarm event queue. When the number of anomaly events awaiting review that continuously enter the queue within the time aggregation window causes the queue length to exceed the capacity threshold, stress sensitivity analysis is initiated.In practice, pressure-sensitive analysis clusters all newly entered anomaly events to be reviewed within the time aggregation window according to their source host identifier and anomaly type. It reads each anomaly event entering the time aggregation window, parses the source host identifier contained in each event, and uses the alarm type label extracted from composite alarm events or the resource indicator type extracted from indicator out-of-bounds events as the anomaly type. A hash mapping table is established with the source host identifier as the primary key and the anomaly type as the subkey. Each anomaly event to be reviewed is categorized into the corresponding entry in the hash mapping table based on its source host identifier and anomaly type. Events under the same entry constitute an initial cluster group. All initial cluster groups are traversed, and groups with the number of events within a preset minimum aggregation threshold are identified as valid cluster groups for subsequent merging to generate aggregated alarm events. In practice, common features are extracted from all anomaly events to be reviewed within the same valid cluster group. These common features include a common source host identifier, anomaly type, and similar time periods. These features are then merged to generate an aggregated alarm event. The aggregated alarm event includes the total number of events in the same group, the first occurrence time of the events within the group, the last occurrence time of the events within the group, and a common description of the relevant resource metrics or log events. The individual anomaly events to be reviewed that have been aggregated are removed from the global alarm event queue, and the generated aggregated alarm event is inserted into the queue. Events are then extracted from the head of the queue in chronological order of their occurrence for subsequent alarm notifications.

[0110] In some embodiments, the initial confidence score is calculated using a weighted composite method. The association confidence of the composite alarm event reflects the reliability of log pattern matching. The out-of-bounds magnitude of the indicator out-of-bounds event is obtained by calculating the ratio of (out-of-bounds value - baseline value) to the standard deviation. The absolute time difference between the composite alarm event and the indicator out-of-bounds event is normalized by dividing by a preset maximum allowable association time difference, transforming it into a dimensionless time correlation coefficient. The smaller the time difference, the closer the time correlation coefficient is to 1. These three factors are assigned different weight coefficients and then weighted and summed to obtain the final score. In some embodiments, the global alarm event queue is implemented using a first-in-first-out queue, sorted according to the generation timestamp of the abnormal events to be reviewed, ensuring that the earliest generated event is processed first. The values ​​of the capacity threshold and the time aggregation window are empirically set based on the system processing capacity and historical alarm frequency.

[0111] It's understandable that the initial confidence score is a quantitative assessment of the credibility of the anomaly event being reviewed at the time of its generation. Subsequent review or processing can refer to this score to determine processing priorities. It's also understandable that the core objective of pressure sensitivity analysis is to aggregate and condense multiple independent anomalies to be reviewed into fewer, more comprehensive aggregated alarm events when a large number of similar alarms are generated in a short period, thereby avoiding alarm storms and improving the readability and processing efficiency of alarm information. The initial confidence score of the anomaly event to be reviewed. This can be formally expressed as:

[0112]

[0113] in: This represents the calculated initial confidence score. Represents the correlation confidence of composite alarm events. The out-of-bounds range of the indicator's out-of-bounds event, its value is That is, out-of-bounds values Reference value of statistical baseline The difference divided by the historical standard deviation , This represents the absolute time difference between a composite alarm event and an indicator out-of-bounds event. This represents the preset maximum allowable time difference for association. , and These are preset weighting coefficients used to adjust the contribution ratio of composite alarm event confidence, indicator out-of-bounds magnitude, and time proximity to the final confidence score.

[0114] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An abnormality detection and alarm pressure sensitive method based on multi-source log correlation analysis, characterized in that, The method includes: Convert log entries of different formats into standardized log events with a unified timestamp, unified event type, and unified source host identifier; From the standardized log events, resource performance index sequences are periodically extracted according to predefined resource performance index extraction rules, and the baseline value of the statistical baseline and the upper and lower limits of the dynamic threshold are calculated for each resource performance index sequence. Using a sliding time window mechanism, the standardized log events are processed in a time-series manner. Within a window, multiple standardized log events that occur simultaneously or successively are matched according to pre-configured event association rules. When the combination conditions of the event association rules are met, a composite alarm event is generated. From the resource performance index sequence, index values ​​that exceed the statistical baseline and dynamic threshold are detected, index out-of-bounds events are generated, and each index out-of-bounds event is spatiotemporally correlated with the composite alarm event. The spatiotemporal correlation is matched based on the temporal proximity of the index out-of-bounds event and the consistency of the source host identifier between the composite alarm event and the composite alarm event. After the spatiotemporal correlation is successfully established, the matched index out-of-bounds event is merged with the composite alarm event to generate an abnormal event to be reviewed. 2.The abnormal detection and alarm pressure sensitive method based on multi-source log correlation analysis according to claim 1, characterized in that, From the standardized log events, resource performance indicator sequences are periodically extracted according to predefined resource performance indicator extraction rules. For each resource performance indicator sequence, a baseline value for statistical baselines and upper and lower limits for dynamic thresholds are calculated, including: For each type of log, the corresponding regular expression pattern is applied to capture numerical resource usage data from the text content of the standardized log events; All captured resource usage data are organized into time series data points according to their source host identifier, resource indicator type, and timestamp to form the resource performance indicator sequence. For each of the resource performance index sequences, the quantile statistical method is used to calculate its multiple quantile values ​​in the past full natural day, and the 50th percentile is used as the benchmark value of the statistical baseline. Based on the baseline value of the statistical baseline and the standard deviation of the fluctuation of the resource performance index sequence in the past statistical period, the upper limit and lower limit of the dynamic threshold are calculated. The upper limit of the dynamic threshold is the baseline value plus a certain number of standard deviations, and the lower limit of the dynamic threshold is the baseline value minus a certain number of standard deviations. 3.The abnormal detection and alarm pressure sensitive method based on multi-source log correlation analysis according to claim 2, characterized in that, The sliding time window mechanism is used to process the standardized log events in a time-series manner. Within a window, multiple standardized log events that occur simultaneously or successively are matched according to pre-configured event association rules. When the combination conditions of the event association rules are met, a composite alarm event is generated, including: Set the window length of the sliding time window, whereby the window length is defined as a fixed duration tracing back from the timestamp of the currently processed standardized log event; Each standardized log event is read sequentially from the streaming log queue and placed into a sliding time window based on the event occurrence time; Query the pre-configured event association rules, which define combination patterns for various event types, the maximum allowed time interval for each event within the combination, and specific attribute conditions that each event must meet; Iterate through all the standardized log events within the current sliding time window to find if there exists an event set whose event type, event attributes, and time difference between events completely match the definition of a certain event association rule. When a set of events that perfectly matches is found, the unique identifiers, event types, and core attributes of all events in the set are aggregated to create a new log entity as the composite alarm event. The composite alarm event contains summary information and association confidence of multiple associated standardized log events, and is assigned an alarm type label that reflects the abnormal scenario represented by the event association rule.

4. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 3, characterized in that, From the resource performance indicator sequence, detect indicator values ​​that exceed the statistical baseline and dynamic threshold, and generate indicator out-of-bounds events, including: Receive data points of the newly collected resource performance index sequence in real time; For each new data point, based on its source host identifier and resource indicator type, the corresponding baseline value of the statistical baseline, the upper limit value and the lower limit value of the dynamic threshold are queried from the stored data; The values ​​of the newly arrived data points are compared with the upper and lower limits of the queried dynamic threshold, respectively; When the value of the newly arrived data point is greater than the upper limit of the dynamic threshold or less than the lower limit of the dynamic threshold, the corresponding data point is determined to have experienced an out-of-bounds event. Record the resource metric type, source host identifier, out-of-bounds value, out-of-bounds direction, timestamp of the out-of-bounds occurrence, and the dynamic threshold exceeded corresponding to the out-of-bounds occurrence. Encapsulate all the recorded information into a single out-of-bounds event and trigger the subsequent spatiotemporal correlation process.

5. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 4, characterized in that, Each of the aforementioned metric out-of-bounds events is spatiotemporally correlated with the composite alarm event. This spatiotemporal correlation is based on matching the temporal proximity and source host identifier consistency between the metric out-of-bounds events and the composite alarm events, including: Whenever an out-of-bounds event of the aforementioned indicator is generated, a time-related interval is immediately defined centered on its timestamp; Retrieve all composite alarm events whose timestamps fall within the time association interval from the event cache, the event cache being used to temporarily store recently generated composite alarm events; From the retrieved composite alarm events, further filter out events whose source host identifier is exactly the same as the source host identifier of the index out-of-bounds event; Calculate the absolute time difference between the timestamp of each selected composite alarm event and the timestamp of the index out-of-bounds event; The absolute time difference is compared with the preset maximum allowable association time difference. If the absolute time difference is less than or equal to the maximum allowable association time difference, it is determined that the composite alarm event and the indicator out-of-bounds event are successfully spatiotemporally associated, and the association relationship between the composite alarm event and the indicator out-of-bounds event is established.

6. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 5, characterized in that, After the spatiotemporal correlation is successful, the matched indicator out-of-bounds event is merged with the composite alarm event to generate an anomaly event to be reviewed, including: Extract the detailed information of the indicator out-of-bounds event, including the type of resource indicator that went out of bounds, the value of the out-of-bounds event, and the direction of the out-of-bounds event; Extract detailed information of the composite alarm event that has been successfully correlated in time and space, including its alarm type label and summary information of the associated standardized log event; Create a new data structure to represent the exception event to be reviewed; The detailed content of the indicator out-of-bounds event, the detailed content of the composite alarm event, and the time difference between the indicator out-of-bounds event and the composite alarm event are all filled into the data structure of the abnormal event to be reviewed. The abnormal event to be reviewed includes the indicator details of the indicator out-of-bounds event, the summary of the associated log event of the composite alarm event, and the relationship between the indicator out-of-bounds event and the composite alarm event. An initial confidence score is calculated for the anomaly event to be reviewed. The confidence score is derived by combining the correlation confidence of the composite alarm event, the out-of-bounds magnitude of the indicator out-of-bounds event, and the time difference between the indicator out-of-bounds event and the composite alarm event.

7. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 6, characterized in that, The upper and lower limits of the dynamic threshold are calculated based on the baseline value of the statistical baseline and the standard deviation of the resource performance index sequence over the past statistical period, including: The coefficient factor of the fluctuation standard deviation is determined based on the predefined sensitivity parameter. From historical storage, obtain all values ​​of the resource performance index sequence within the most recent complete statistical period, and calculate the standard deviation of the corresponding set of values ​​as the fluctuation standard deviation; The upper limit of the dynamic threshold is obtained by adding the product of the baseline value of the statistical baseline and the standard deviation of the fluctuation multiplied by the coefficient factor. The lower limit of the dynamic threshold is obtained by subtracting the product of the baseline value of the statistical baseline and the standard deviation of the fluctuation multiplied by the coefficient factor.

8. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 7, characterized in that, The absolute time difference between the timestamp of each composite alarm event selected through calculation and the timestamp of the indicator out-of-bounds event includes: Read the timestamp recorded in the index out-of-bounds event and record it as the first timestamp; Read the timestamp recorded in one of the filtered composite alarm events and record it as the second timestamp; Calculate the difference between the second timestamp and the first timestamp; The absolute time difference is obtained by taking the absolute value of the calculated difference. The calculated absolute time difference is compared with a preset maximum allowable associated time difference, which is dynamically configured according to the resource indicator type of the indicator out-of-bounds event and the alarm type label of the composite alarm event.

9. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 8, characterized in that, Create a new data structure as the exception event to be reviewed, including: Define a structure that encapsulates all information about the exception event to be reviewed; A first field is set in the structure to store the globally unique identifier of the abnormal event to be reviewed. The globally unique identifier is generated by combining the timestamp, the source host identifier, and the event type code. A second field is set in the structure to store the detailed content of the indicator out-of-bounds event. The detailed content specifically includes the type of resource indicator that went out of bounds, the out-of-bounds value, the out-of-bounds direction, the dynamic threshold that was exceeded, and the timestamp of the indicator out-of-bounds event. A third field is set in the structure to store the detailed content of the composite alarm event. The detailed content specifically includes the alarm type label of the composite alarm event, the list of unique identifiers of the associated standardized log events, the core attribute summary of the standardized log events, and the generation timestamp of the composite alarm event. A fourth field is set in the structure to store the association information, which includes the absolute time difference between the indicator out-of-bounds event and the composite alarm event, and the timestamp of the association establishment. A fifth field is set in the structure to store the initial confidence score; Based on the definition of the structure, a new exception event object to be reviewed is instantiated in memory, and the first, second, third, fourth and fifth fields are filled with the corresponding data to complete the creation of the exception event to be reviewed.

10. The anomaly detection and alarm pressure-sensitive method based on multi-source log correlation analysis according to claim 9, characterized in that, The method also includes a step of performing alarm and pressure-sensitive processing on the abnormal events to be reviewed: Establish a global alarm event queue to store all generated abnormal events pending review; Configure a capacity threshold and a time aggregation window for the global alarm event queue; Monitor the length of the global alarm event queue. When the number of consecutive abnormal events to be reviewed entering the queue within the time aggregation window causes the queue length to exceed the capacity threshold, initiate pressure sensitivity analysis. The pressure sensitivity analysis clusters all newly entered anomaly events to be reviewed within the time aggregation window according to their source host identifier and anomaly type. For all abnormal events to be reviewed within the same group, their common features are extracted and merged to generate an aggregated alarm event. The aggregated alarm event includes the total number of events in the same group, the first occurrence time, the last occurrence time, and a common description of the resource indicators or log events involved. Remove the individual aggregated exception events to be reviewed from the global alarm event queue, insert the generated aggregated alarm event into the queue, and then extract the events from the head of the queue in the order of their occurrence for subsequent alarm notifications. The pressure sensitivity analysis clusters all newly entered anomaly events to be reviewed within the time aggregation window according to their source host identifier and anomaly type, including: Read each pending exception event that enters within the time aggregation window; Parse the source host identifier contained in each anomaly event to be reviewed, and the alarm type label extracted from the composite alarm event or the resource indicator type extracted from the indicator out-of-bounds event; Establish a hash mapping table with the source host identifier as the primary key and the exception type as the subkey; Each anomaly event to be reviewed is categorized into the corresponding entry in the hash mapping table based on its source host identifier and anomaly type. Events under the same entry constitute an initial cluster group. Iterate through all initial cluster groups. For groups whose number of events reaches the preset minimum aggregation threshold, identify them as valid cluster groups for subsequent merging to generate aggregated alarm events.

Images