Abnormality detection method, device, system and medium based on log entropy features

Abstract

The application discloses an abnormality detection method, device and system based on log entropy features and a medium, and belongs to the field of data processing. The method comprises the following steps: obtaining a newly obtained entropy feature, wherein the newly obtained entropy feature is an entropy feature of a binary data block of a newly appeared log; determining an abnormality detection reference range based on the entropy features of the binary data blocks of the logs within a historical time length; in the case where the relationship between the newly obtained entropy feature and the abnormality detection reference range meets an abnormality detection condition, locking the binary data block corresponding to the newly obtained entropy feature and generating abnormality event data; constructing a prompt word based on the locked binary data block and the abnormality event data, and calling a large language model to infer and output abnormality root cause report data according to the prompt word. According to the embodiment of the application, the abnormality recognition rate can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to an anomaly detection method, apparatus, system and medium based on log entropy characteristics. Background Technology

[0002] In the operation and maintenance monitoring system of a business system, logs can record key events, error propagation, state changes, and request context information during the operation of the business system. As the core carrier of the business system's operating status, logs provide a detailed description of the internal state and business operation process of the business system. The monitoring and analysis of logs rely on keyword rule matching. However, in complex business environments, the keyword rule matching method relies on parsing the text content of the logs, which cannot detect non-textual anomalies or anomalies without displayed errors, resulting in missed anomaly identification and reducing the anomaly identification rate. Summary of the Invention

[0003] This application provides an anomaly detection method, apparatus, system, and medium based on log entropy features, which can improve the anomaly recognition rate.

[0004] In a first aspect, embodiments of this application provide an anomaly detection method based on log entropy features, comprising: acquiring newly obtained entropy features, wherein the newly obtained entropy features are the entropy features of binary data blocks of newly appearing logs; determining an anomaly detection benchmark range based on the entropy features of binary data blocks of logs within a historical time period; locking the binary data block corresponding to the newly obtained entropy features and generating anomaly event data when the relationship between the newly obtained entropy features and the anomaly detection benchmark range satisfies the anomaly detection condition; constructing prompt words based on the locked binary data blocks and anomaly event data, and calling a large language model to infer and output anomaly root cause report data based on the prompt words.

[0005] Secondly, embodiments of this application provide an anomaly detection device based on log entropy features, comprising: an entropy feature acquisition module, used to acquire newly obtained entropy features, wherein the newly obtained entropy features are the entropy features of binary data blocks of newly appearing logs; a benchmark determination module, used to determine an anomaly detection benchmark range based on the entropy features of binary data blocks of logs within a historical time period; an anomaly determination module, used to lock the binary data block corresponding to the newly obtained entropy features and generate anomaly event data when the relationship between the newly obtained entropy features and the anomaly detection benchmark range satisfies the anomaly detection conditions; and a root cause analysis module, used to construct prompt words based on the locked binary data blocks and anomaly event data, and call a large language model to infer and output anomaly root cause report data based on the prompt words.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the anomaly detection method based on log entropy features of the first aspect.

[0007] Fourthly, embodiments of this application provide an anomaly detection system based on log entropy features, comprising: an edge node device, configured to, in response to the occurrence of new logs, acquire the binary data block of the newly acquired logs and the entropy features of the binary data block, and upload the newly acquired entropy features to a central processing platform; and a central processing platform, communicatively connected to the edge node device, configured to execute the anomaly detection method based on log entropy features of the first aspect.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the anomaly detection method based on log entropy features of the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the anomaly detection method based on log entropy features of the first aspect.

[0010] This application provides an anomaly detection method, apparatus, system, and medium based on log entropy features. When a new log appears, the entropy features of the binary data blocks in the newly appearing log are obtained. Based on the entropy features of binary data blocks in logs over a historical period, an anomaly detection benchmark range suitable for the current situation can be determined. By analyzing the relationship between the newly obtained entropy features and the anomaly detection benchmark range, anomalies are identified, the corresponding binary data blocks are locked, and anomaly event data characterizing the anomaly is generated. Leveraging the reasoning capabilities of a large language model, and based on prompt words constructed from the locked binary data blocks and anomaly event data, anomaly root cause report data characterizing the root cause of the anomaly can be obtained. Entropy features can reflect the regularity and information density of data in binary data blocks, without relying on the semantics of the logs. Entropy features can identify anomalies caused by business systems entering infinite loops, binary data flowing into logs, etc., enabling more comprehensive anomaly detection without semantic dependence. This fills the technical blind spots of anomaly detection relying on semantic content parsing, reduces missed anomaly identification, and improves the anomaly detection rate. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1A schematic diagram of an anomaly detection system based on log entropy features provided in an embodiment of this application; Figure 2 A flowchart illustrating an anomaly detection method based on log entropy features provided in an embodiment of this application; Figure 3 A schematic diagram illustrating an example of an anomaly detection method based on log entropy features provided in an embodiment of this application; Figure 4 A schematic diagram illustrating an example of an anomaly detection system based on log entropy features provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an anomaly detection device based on log entropy features provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the purpose, technical solution, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of the details in these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or necessarily used such a solution.

[0014] In the operation and maintenance monitoring system of a business system, logs record key events, error propagation, state changes, and request context information during the system's operation. As the core carrier of the business system's operational status, logs provide a detailed description of the system's internal state and business operation processes. Log monitoring and analysis rely on keyword rule matching. However, in complex business environments, keyword rule matching relies on parsing the log's text content, failing to detect non-textual anomalies or anomalies without displayed errors, leading to missed anomaly identification and reduced anomaly detection rate. For example, if the business system enters an infinite loop, repeatedly printing INFO (Information) level logs, this repeated printing of INFO logs, which only involves content repetition without semantic errors, cannot be detected using keyword rule matching. Similarly, when binary data flows into the logs, since binary data cannot be decoded using UTF-8, keyword rule matching cannot find parsable text content. Furthermore, binary data is difficult for operations personnel to interpret manually, requiring additional information from other aspects of the business to identify the anomaly. Furthermore, the amount of log data can surge when an anomaly occurs. If template clustering or deep learning inference is used to detect anomalies, the O(n) time complexity of template clustering is significantly reduced. 2 The complexity of O(n) or deep learning inference is O(n) 3 The complexity of the system can lead to a "computing power inversion" phenomenon, causing the detection system to crash before the business system and making it unable to cope with traffic surges. In addition, the keyword rule matching method lacks the ability to automatically close the loop in analyzing the root causes of anomalies. After an anomaly is detected, logs need to be downloaded and read manually to determine the root cause. If the logs contain binary data or garbled content, the operations and maintenance personnel cannot read them, making it difficult to determine the root cause of the anomaly.

[0015] This application provides an anomaly detection method, apparatus, device, system, medium, and program product based on log entropy features. It can achieve full-coverage anomaly detection without semantic dependencies, identify infinite loop anomalies and binary data leakage anomalies, and its detection capability is not limited to log formats, but is compatible with multiple log formats, filling the technical blind spot of anomaly detection based on text content parsing. It can achieve lightweight feature extraction with microsecond-level latency, thereby solving the "computing power inversion" problem in high-load scenarios and can effectively cope with traffic surges. After determining the binary data block corresponding to the anomaly, it can construct prompt words and realize automated closed-loop analysis of the anomaly root cause through a large language model, reducing the rate of manual intervention and improving the recognition rate of the anomaly root cause.

[0016] The following describes the anomaly detection method, apparatus, equipment, system, medium, and program products based on log entropy features provided in this application.

[0017] To facilitate understanding, we will first provide a brief introduction to anomaly detection systems based on log entropy features. Figure 1 This is a schematic diagram of an anomaly detection system based on log entropy features provided in an embodiment of this application, as shown below. Figure 1 As shown, the anomaly detection system based on log entropy features may include an edge node device 11 and a central processing platform 12.

[0018] Edge node device 11 can be implemented as a business server independent of the business system, located on the business server side of the business system; edge node device 11 can also be implemented as a business server of the business system, implemented by setting an agent in the business server. This agent can be deployed as a sidecar container and co-deployed with the business Pod of the business system, or run as an independent DaemonSet. The hardware carrier of edge node device 11 can include, but is not limited to, x86_64 / AArch64 architecture servers, without relying on a graphics processing unit (GPU). Edge node device 11 can monitor logs, respond to the occurrence of new logs, obtain the binary data blocks of the newly obtained logs and their entropy characteristics, and upload the entropy characteristics of the binary data blocks to the central processing platform 12. The entropy characteristics of the newly obtained log binary data blocks can be referred to as the newly obtained entropy characteristics. The specific content of the steps in the anomaly detection method based on log entropy characteristics executed by edge node device 11 will be explained in the following section on anomaly detection methods based on log entropy characteristics, and will not be repeated here.

[0019] The central processing platform 12 can be a remote platform, implemented through one or more electronic devices. The number and type of electronic devices implementing the functions of the central processing platform 12 are not limited here. The central processing platform 12 communicates with the edge node devices 11 and can interface with multiple edge node devices 11. It has the ability to process entropy features transmitted from multiple edge node devices 11 in parallel and can perform anomaly detection based on entropy features. The specific details of the steps in the anomaly detection method based on log entropy features executed by the central processing platform 12 will be explained later in the section on anomaly detection methods based on log entropy features, and will not be repeated here.

[0020] This application provides an anomaly detection method based on log entropy features, which can be applied to scenarios where anomalies are detected based on logs. This log entropy-based anomaly detection method can be executed by a central processing platform, a log entropy-based anomaly detection system, etc. The central processing platform can be implemented through an anomaly detection device based on log entropy features, electronic equipment, etc., and is not limited thereto. Figure 2 A flowchart of an anomaly detection method based on log entropy features provided in an embodiment of this application is shown below. Figure 2 As shown, the anomaly detection method based on log entropy features may include steps S201 to S204.

[0021] In step S201, the newly obtained entropy features are acquired.

[0022] Edge node devices, configured on the business server side of the business system, can respond to newly appearing logs and obtain the binary data blocks and entropy characteristics of these blocks. Here, "logs" refers to the logs of the business system. The binary data blocks of newly appearing logs are obtained by fragmenting the binary data of the newly appearing logs. The edge node device can obtain the binary data of the newly appearing logs, fragment the binary data to obtain binary data blocks (Chunks), and obtain the entropy characteristics of each binary data block. The newly obtained entropy characteristics are the entropy characteristics of the binary data blocks of the newly appearing logs. The edge node device can monitor the logs and, when a new log appears, trigger actions such as obtaining the binary data of the new log, fragmenting it to obtain binary data blocks, and calculating the entropy characteristics of the binary data blocks. For example, the edge node device can determine whether a new log has appeared by monitoring log file write events. When a log file write event is detected, the device can use the `read()` system call file descriptor to read data from the kernel's I / O buffer to obtain the binary data of the newly appearing log. Reading data from the kernel's I / O buffer can be done in non-blocking mode (O_NONBLOCK mode), but is not limited to this. For example, edge node devices can determine the presence of new logs by monitoring data arrival events in the standard output (StdOut) pipes of containers in the business system. Upon detecting a data arrival event, they can subscribe to the raw binary stream from the Docker daemon's log pipe interface, i.e., the Docker Engine API, to obtain the binary data of the new logs. The binary data of the logs can be in any binary format; there are no restrictions here. However, it should be noted that the obtained log binary data is the raw byte stream of the logs, without transcoding or decoding such as UTF-8 or GBK. The sampling interval for edge node devices to collect log binary data can be set by time or byte length; there are no restrictions here. For example, sampling can be in a fixed-time mode with a sampling interval of 1 second; or sampling can be in a fixed-byte mode with a sampling interval of 1MB. Edge node devices can set the maximum concurrent sampling handles, i.e., pre-setting the thread pool capacity for threads collecting log binary data. The specific setting can be based on the scenario, requirements, experience, etc., and is not limited here. For example, the maximum concurrent sampling handles can be 1000. If concurrent sampling handles are exhausted and new binary data arrives, the new binary data is discarded directly, and the number of discards is recorded for reporting. The size of the log binary data block can be set according to the scenario, requirements, experience, etc. For example, the size of the log binary data block can be 4KB. If the size of the last binary data block of the collected log binary data fragment is less than 4KB, the size of this last binary data block can be the actual length without padding.The timeout period for binary data blocks can be set. This timeout is the maximum waiting time for binary data that is insufficient to be divided into separate binary data blocks. It can be set based on the scenario, requirements, and experience, and is not limited here. For example, the timeout period for a binary data block could be 500 milliseconds. In some examples, the binary data block has a metadata header. The metadata header may include, but is not limited to, a data block identifier, a timestamp, and a business process identifier. Furthermore, the metadata header may also include, but is not limited to, the data block size and a data block checksum. The data block identifier is used to identify the binary data block; for example, the data block identifier could be a data block ID. The timestamp represents the generation time of the binary data in the binary data block. The business process identifier is used to identify the business process associated with the binary data in the binary data block. The data block checksum is used to verify the binary data block; for example, the data block checksum could be a Cyclic Redundancy Check (CRC) code. For example, the format of the metadata header for a binary data block could be as follows: message RawChunk { string chunk_id = XXX; int64 timestamp_ns = XXXXXX; int32 source_pid = XXXXX; bytes raw_data = 4096; uint32 checksum = XXXXXXX; } Among them, string chunk_id represents the data block identifier, int64 timestamp_ns represents the nanosecond-level timestamp, int32 source_pid represents the business process identifier, bytes raw_data represents the data block size in bytes, and uint32 checksum represents the CRC code.

[0023] A preset entropy feature algorithm can be invoked to obtain the entropy feature of binary data blocks based on the binary data blocks. The binary data blocks obtained from binary data fragmentation of the log can be stored in a lock-free circular buffer (RingBuffer) local to the edge node device. The sampling delay from obtaining the binary data block to its entry into the lock-free circular buffer can be less than or equal to 1 millisecond, and the Central Processing Unit (CPU) utilization rate can be less than or equal to 1%. Each time the entropy feature of a binary data block is generated, the binary data block can be deleted from the lock-free circular buffer. The capacity of the lock-free circular buffer can be preset; for example, setting the capacity to 100 binary data blocks means that if the generation of the entropy feature of a binary data block fails, the lock-free circular buffer can buffer a maximum of 100 binary data blocks. If the lock-free circular buffer is full, a removal and discard policy can be enabled to discard the oldest buffered binary data block. If the business process corresponding to the binary data exits during the process of collecting binary data from logs, it can be retried after a period of time, such as retrying after 5 seconds of sleep. If the connection still fails after multiple retries, such as after 3 retries, the data source of the binary data can be marked as offline.

[0024] The entropy characteristic of a binary data block can characterize the regularity and information density of the data within the block. Entropy characteristics may include, but are not limited to, the compression ratio obtained by compressing the binary data block using a preset compression algorithm, or the information entropy obtained by performing calculations on the binary data block using an entropy algorithm.

[0025] In some examples, the default compression algorithm can be LZ4 compression, the compression level of LZ4 can be 1, and the application programming interface (API) for calling LZ4 compression can be LZ4_compress_default(input, output, input_len, max_output_len). The compression speed of LZ4 is approximately 500MB / s, the decompression speed is greater than 1GB / s, and the computational complexity O(n) increases linearly with the length of the binary data block. The processing latency of a single binary data block of the compression algorithm can be less than or equal to 0.8 milliseconds, the throughput of the compression algorithm can be greater than or equal to 5000 blocks / second, and the memory usage can be less than or equal to 64MB. For example, the compression ratio can be calculated according to the following formula (1): (1) in, This refers to the compression ratio; This represents the size of the binary data block before compression. This refers to the size of the compressed binary data block.When the binary data block is empty (==0), the compression ratio can be set to 1. A higher compression ratio indicates that the binary data block is easier to compress, has stronger regularity, and lower information density; a lower compression ratio indicates that the data is harder to compress, has lower regularity, higher randomness, and higher information density. If compression of a binary data block using a preset compression algorithm fails, it can be retried using a different compression algorithm. If the retry fails, the compression ratio can be set to a preset maximum compression ratio, such as 999.99, indicating an anomaly in the binary data block. Specifically, this indicates a discrete anomaly. For example, the LZ4 algorithm can be used to compress the binary data block first; if this fails, the Snappy algorithm can be used for a retry. Binary data blocks can be compressed block by block or in batches; this is not limited here. A compression timeout can be set, which is the duration for compressing a single binary data block. If the compression timeout for a single binary data block exceeds the timeout, the compression ratio can be set to the preset maximum compression ratio, indicating an anomaly in the binary data block, and marking it as an abnormal binary data block. The compression timeout can be set based on scenarios, requirements, experience, etc., and is not limited here. For example, a compression timeout of 50 milliseconds is possible. During the compression of binary data blocks, if insufficient memory occurs, the number of concurrent compression threads can be limited. A maximum thread limit can be set. When the number of compression threads exceeds the maximum thread limit, the tasks corresponding to the excess threads can enter a queue to wait. The maximum thread limit can also be set based on scenarios, requirements, experience, etc., and is not limited here. For example, the maximum thread limit can be 1.5 times the number of CPU cores. In some examples, where the entropy feature includes the compression ratio, binary data blocks have quality labels. These quality labels are obtained based on the correlation between the compression ratio and information entropy of some binary data blocks, which are extracted from multiple binary data blocks in newly appearing logs. Quality tags characterize the quality of binary data blocks. Based on the quality tags, it can be determined whether a binary data block and its compression ratio are suitable for subsequent anomaly detection processes. A subset of binary data blocks can be extracted from multiple binary data blocks in newly emerging logs. The information entropy of this subset is calculated. The quality tags for these multiple binary data blocks are obtained based on the correlation between their compression ratio and information entropy. Cross-validation can be performed using the correlation between the compression ratio and information entropy of this subset of binary data blocks. This cross-validation is performed on samples from multiple binary data blocks in the newly emerging logs, not on every single binary data block.The correlation between the compression ratio and information entropy of a binary data block can characterize whether the two are positively or negatively correlated, and the degree of positive or negative correlation. For example, the correlation between the compression ratio and information entropy of a binary data block can be reflected by the correlation coefficient. In this embodiment, the compression ratio and information entropy should theoretically have a strong negative correlation. If the correlation coefficient between the compression ratio and information entropy of a binary data block is less than or equal to a first correlation threshold, the quality label of the binary data block and the quality label of the same batch of binary data blocks can be normal. The first correlation threshold is less than 0 and can be set according to the scenario, requirements, and experience. For example, the first correlation threshold can be -0.8. If the correlation coefficient between the compression ratio and information entropy of a binary data block is greater than the first correlation threshold and less than the second correlation threshold, it indicates that the compression ratio and information entropy of the binary data block have a weak negative or weak positive correlation. It can be considered that the binary data block and the quality label of the same batch of binary data blocks are normal. The binary data block may contain anomalies. The quality label of this binary data block, along with the quality labels of other binary data blocks in the same batch, can be classified as "Suspicious." The second correlation threshold is greater than 0 and can be set according to the scenario, requirements, and experience; for example, a strong negative correlation threshold could be 0.8. If the correlation coefficient between the compression ratio and information entropy of the binary data block is greater than or equal to the second correlation threshold, it indicates a strong positive correlation between the compression ratio and information entropy of this binary data block. This binary data block, along with other binary data blocks in the same batch, can be considered invalid binary data blocks, and their quality labels, along with those of other binary data blocks in the same batch, can be classified as "Invalid." Binary data blocks with normal quality labels and their compression ratios can participate in subsequent anomaly detection processes. The situation where binary data blocks with "Suspicious" quality labels may be caused by abnormal input from the compression algorithm. It should be noted that in this cross-validation process, information entropy is used as auxiliary information for cross-validation, not as an entropy feature. The impact of the above cross-validation on the entire anomaly detection process is very small, causing a delay of less than or equal to 0.1 milliseconds, which can be ignored.

[0026] In some examples, the entropy algorithm may include, but is not limited to, the Shannon entropy algorithm, and the information entropy can be calculated according to the following equations (2) and (3): (2) (3) in, For the i-th byte Frequency of occurrence in binary data blocks; The information entropy of a binary data block; For the i-th byte The total number of times it appears in a binary data block; This represents the size of the binary data block. The smaller the information entropy of the binary data block, the easier it is to compress, the stronger its regularity, and the lower its information density; the larger the information entropy, the more difficult it is to compress, the lower its regularity, the higher its randomness, and the higher its information density.

[0027] Entropy features, besides including compression ratio or information entropy, may also include, but are not limited to, data block identifiers, timestamps, entropy feature processing time, and entropy feature algorithm identifiers. When the entropy feature includes compression ratio, the entropy feature processing time can be the compression duration, and the entropy feature algorithm identifier can be the compression algorithm identifier. When the entropy feature includes information entropy, the entropy feature processing time can be the information entropy calculation time, and the entropy feature algorithm identifier can be the entropy algorithm identifier. For example, the entropy feature can be in JSON format, and can also be called an EntropyFeature. Taking the entropy feature including compression ratio as an example, the entropy feature can be implemented as the following data structure: json { "chunk_id": "XXX", "timestamp_ns": XXXXXX, "compression_ratio": 4.56, "compress_time_us": 45, "algorithm": "LZ4 } Where `chunk_id` represents the data block identifier; `timestamp_ns` represents the timestamp; `compression_ratio` represents the compression ratio; `compress_time_us` represents the compression time; and `algorithm` represents the compression algorithm identifier. Entropy features can be stored in the local time series database in InfluxDB Line Protocol format. The time series database can temporarily cache entropy features within the most recent time period, for example, entropy features within the most recent 10 minutes.

[0028] After generating entropy features, the edge node devices upload the entropy features to the central processing platform. The central processing platform then obtains the entropy features and executes subsequent processes.

[0029] In step S202, the anomaly detection baseline range is determined based on the entropy characteristics of binary data blocks in the logs over a historical period.

[0030] The central processing platform can periodically trigger the process of determining the anomaly detection baseline range, or it can trigger the process of determining the anomaly detection baseline range when it receives entropy features uploaded by edge node devices. If the process of determining the anomaly detection baseline range is triggered periodically, the periodic duration of the process can be, but is not limited to, 5 minutes, and can be set according to the scenario, requirements, experience, etc.

[0031] The historical duration can be the most recent time period, and its specific value can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the historical duration can be the past 7 days. The entropy features of binary data blocks obtained within the historical duration can be stored in a local time series database. The entropy features of binary data blocks that meet the requirements within the historical duration can be retrieved from the local time series database. Based on the entropy features that meet the requirements within the historical duration, an anomaly detection benchmark range is determined. This anomaly detection benchmark range will change with time and newly obtained entropy features; that is, the anomaly detection benchmark range is a dynamically changing range that can adapt to newly obtained entropy features. The anomaly detection benchmark range can be used to identify anomalies. By comparing the newly obtained entropy features with the anomaly detection benchmark range, it can be determined whether an anomaly has occurred and the type of anomaly.

[0032] In some embodiments, each time a new entropy feature is acquired, an entropy feature that is temporally correlated with the new entropy feature over a historical period can be obtained. A static baseline range can be obtained through robust statistical processing. By sensing the new entropy feature, a real-time drift amount that reflects the changing trend of the entropy feature under the current situation can be determined. The static baseline range can be corrected using the real-time drift amount to obtain an anomaly detection baseline range that conforms to the current actual situation. By the relationship between the new entropy feature and the anomaly detection baseline range, it can be determined whether an anomaly has occurred and the type of anomaly.

[0033] In other examples, each time a new entropy feature is obtained, a pre-trained isolated forest model can be used to obtain an anomaly score that reflects the deviation between the new entropy feature and the entropy feature over a historical period. The average value of the entropy features over a historical period is used to determine the anomaly detection baseline range. By combining the relationship between the new entropy feature and the anomaly detection baseline range with the anomaly score, it can be determined whether an anomaly has occurred and the type of anomaly.

[0034] In step S203, if the relationship between the newly obtained entropy feature and the anomaly detection benchmark range satisfies the anomaly detection condition, the binary data block corresponding to the newly obtained entropy feature is locked and anomaly event data is generated.

[0035] Anomaly detection conditions include those used to determine the occurrence and type of anomalies. Multiple anomaly detection conditions are possible, each corresponding to a different anomaly type. If the relationship between the newly obtained entropy feature and the anomaly detection baseline range satisfies the anomaly detection conditions, it indicates that the business corresponding to the newly appearing binary data block of the log has experienced an anomaly. The anomaly type can also be determined based on the anomaly detection conditions satisfied by the relationship between the newly obtained entropy feature and the anomaly detection range.

[0036] Locking the binary data block corresponding to the newly obtained entropy feature specifically refers to locating the binary data block based on the data block identifier in the entropy feature and copying and retaining the binary data block to preserve the original binary data. Anomaly event data can characterize the business anomaly corresponding to the newly appearing log binary data block. In some examples, anomaly event data may include an anomaly event identifier, anomaly type, and the data block identifier of the locked binary data block. Anomaly event data may also include, but is not limited to, one or more of the following: anomaly severity, entropy feature, timestamp, drift cause, anomaly detection baseline range, number of exceedances, and the original data of the locked binary data block. The anomaly event identifier is used to identify the anomaly event; for example, the anomaly event identifier may be an anomaly event ID. In some examples, the anomaly type may include redundant anomalies (i.e., RedundancyAnomaly) and discrete anomalies (i.e., DispersionAnomaly). Redundancy anomalies characterize an increase in repetitive patterns of data in the logs, indicating that the business system's operational trajectory is becoming monotonous, mechanical, and looping, indicating an anomaly. The root causes of this redundant anomaly may include, but are not limited to, the business system entering an infinite loop, the business system frequently retrying abnormally, the business system outputting heartbeat or keep-alive signals at frequencies far exceeding the normal range, and receiving templated brute-force attacks. Discrete anomalies are characterized by an increase in high-entropy randomized data in the logs, indicating that the business system's operational trajectory is becoming more randomized, disordered, and unpredictable, indicating an anomaly. The root causes of this discrete anomaly may include, but are not limited to, data leakage, data corruption, accidentally writing binary data to the logs, hardware damage leading to the reading of random values, and hardware damage leading to the return of uninitialized buffered data. Anomaly severity can characterize the severity of the anomaly. The severity can be reflected by the degree of deviation of the newly obtained entropy feature from the entropy feature within the historical time period. When an anomaly has been confirmed, the higher the deviation of the newly obtained entropy feature from the entropy feature within the historical time period, the higher the severity of the anomaly; the lower the deviation, the lower the severity. Anomaly severity may include, but is not limited to, high, medium, and low levels. The cause of drift characterizes the reason for the drift of the determined anomaly detection benchmark range relative to the static benchmark range. For example, the cause of drift may include, but is not limited to, business system expansion, business system downsizing, and normal fluctuations in business system performance. In some cases, to avoid misjudging normal fluctuations in entropy features as anomalies, an anomaly can be determined only when newly obtained entropy features repeatedly exceed the anomaly detection benchmark range or repeatedly hit the anomaly detection benchmark range. The number of times the limit is exceeded can be the number of times the newly obtained entropy features consecutively exceed the anomaly detection benchmark range, or the number of times the newly obtained entropy features consecutively hit the anomaly detection benchmark range. The original data of the locked binary data block can be Hex data.For example, abnormal event data can also be called an AnomalyEvent structure, which can be in JSON format. Taking entropy features including compression ratio as an example, the format of abnormal event data can be as follows: json { "anomaly_id": XXXXX, "type": "REDUNDANCY", "severity": "HIGH", "trigger_chunk_id": XXXXXX, "cr_value": 52.4, "baseline_upper": 7.1, "baseline_lower": 2.5, "timestamp_ns": XXXX, "continuous_count": 3, "locked_data_hex": "5B494E464F5D205072..." } Here, `anomaly_id` is the anomaly event identifier; `type` is the anomaly type, with `REDUNDANCY` indicating a redundant anomaly; `severity` is the anomaly severity, with `HIGH` indicating a high severity; `trigger_chunk_id` is the block identifier of the locked binary data block; `cr_value` is the compression ratio; `baseline_upper` is the upper limit of the anomaly detection baseline range; `baseline_lower` is the lower limit of the anomaly detection baseline range; `timestamp_ns` is the timestamp; `continuous_count` is the number of times the limit is exceeded; and `locked_data_hex` is the raw binary data of the locked binary data block. To reduce resource consumption, the raw data of the locked binary data block can be converted to hexadecimal data. After obtaining the anomaly event data, it can be stored in an anomaly queue, which can be, but is not limited to, a RedisStream, and simultaneously recorded in the log file based on the central processing platform.

[0037] It is important to note that the same data block identifier will only trigger the generation of abnormal event data once within a preset trigger duration to avoid duplicate alarms and triggering an alarm information storm. Abnormal event data is used in subsequent processes to obtain abnormal root cause report data. If problems occur during the generation of abnormal root cause report data and the abnormal event data cannot be consumed, the abnormal event data can be retained in a local retry queue and retried using an exponential backoff method. Here, retry refers to attempting to generate abnormal root cause report data again based on the abnormal event data. The retry interval for the exponential backoff method can be 1 second, 2 seconds, 4 seconds, ..., but the retry interval must not exceed 60 seconds. The local retry queue can retain multiple recent abnormal event data, such as a maximum of 100 recent abnormal event data. In this embodiment, the delay from receiving the entropy feature to generating abnormal event data can be less than or equal to 1 millisecond, and the delay from the generation of abnormal event data to triggering an alarm can be less than or equal to 100 milliseconds. This process consumes very little time, improving the timeliness of anomaly detection.

[0038] In step S204, based on the locked binary data block and the abnormal event data, a prompt word is constructed, and the large language model is called to infer and output the abnormal root cause report data according to the prompt word.

[0039] Clue words can be used to guide large language models in reasoning and output anomaly root cause report data. Clue words can include, but are not limited to, roles and tasks. Task descriptions can include, but are not limited to, the raw data of binary data blocks, at least a portion of the anomaly event data, and anomaly root cause analysis questions. Clue word templates can be pre-set; different anomaly types can correspond to different prompt word templates. By filling the prompt word template with the raw data of the locked binary data block and at least a portion of the anomaly event data, prompt words can be constructed.

[0040] For example, if the anomaly is determined to be a redundant anomaly, the suggested message could be as follows: "Role: You are a senior Java / Python application operations and maintenance expert, skilled at analyzing duplicate log patterns to infer code defects."

[0041] Task: Analyze the following hexadecimal encoded log fragment. The fragment has extremely low information entropy (compression ratio = {cr_value}, normal range 2-8), indicating that the content is highly repetitive.

[0042] Hexadecimal data (first 512 bytes): {locked_data_hex} Please answer: 1. What is the recurring pattern? (e.g., "Process checking...", Exception stack trace) 2. What are the possible flaws in the code logic? (e.g., infinite loop while(true), endless recursion) 3. Suggested troubleshooting steps (please specify code files / configuration item checkpoints)".

[0043] For example, if the anomaly is determined to be a discrete anomaly, the suggested message could be as follows: "Role: You are a system security expert, skilled in analyzing the characteristics of binary garbled / encrypted data."

[0044] Task: Analyze the following hexadecimal encoded data segment. This segment has extremely high information entropy (compression ratio = {cr_value}, close to 1.0 without compression), and may be garbled text, encrypted data, or a memory dump.

[0045] Hexadecimal data (first 512 bytes): {locked_data_hex} Byte distribution characteristics (Top 10 most frequent bytes): {byte_freq_analysis} Please answer: 1. What is the most likely content type? (Garbled text / UTF8 decoding error / encrypted ciphertext / binary image / coreedump) 2. Possible causes of this issue? (Character set configuration error / Incorrect encryption module key / Out-of-bounds memory write) 3. Safety risk level assessment (no risk / low / medium / high) and handling recommendations. The hexadecimal data in the above prompt is hexadecimal data converted from the original data of the locked binary data block. When this prompt is input into a large language model, the model can infer based on the information in the prompt, thus providing root cause report data containing information such as recurring patterns, code logic defects, troubleshooting steps, content type, cause of the anomaly, and security risk level.

[0046] Anomaly root cause report data characterizes the underlying cause of detected anomalies. In some examples, anomaly root cause report data may include a report identifier, anomaly event identifier, anomaly type, and root cause result. Anomaly root cause report data may also include, but is not limited to, one or more of the following: large language model identifier, confidence level, risk level, recommended actions, data generation time, and token usage. The report identifier identifies the anomaly root cause report data; for example, the report identifier may be the ID of the anomaly root cause report data. The root cause result is information characterizing the root cause that caused the anomaly. The large language model identifier identifies the large language model from which the anomaly root cause report data originated. The confidence level is the confidence level of the root cause result output by the large language model. The risk level is the risk level corresponding to the anomaly. Recommended actions include suggested actions to be taken in response to the anomaly caused by the root cause result. The data generation time is the time consumed in generating the anomaly root cause report data. The token usage refers to the amount of tokens consumed in obtaining the root cause report data. This can include the tokens consumed for input and the tokens consumed for output. The tokens consumed for input are the tokens for the prompt words, and the tokens consumed for output are the tokens consumed by the large language model to generate the output content. For example, the root cause report data can also be called a RootCauseReport structure, which can be in JSON format. The format of the root cause report data can be as follows: json { "report_id":XXXX, "anomaly_id": XXXXX, "diagnosis_type": "REDUNDANCY_ANALYSIS", "llm_model": "DeepSeek-Coder-V2", "diagnosis_result": "A highly repetitive pattern was detected: '[INFO] User loginattempt: user_id=XXXXX' is printed every 5ms. The possible cause is an infinite loop in the user login API. It is recommended to check the retry logic of the AuthController.login() method." "confidence_score": 0.87, "risk_level": "HIGH", "suggested_actions": ["Check line 145 of AuthController.java", "Confirm the while loop termination condition"], "generation_time_ms": 1250, "token_usage": { "prompt": 856, "completion": 234} } Wherein, `report_id` is the report identifier; `anomaly_id` is the anomaly event identifier; `diagnosis_type` is the anomaly type; `llm_model` is the large language model identifier; `diagnosis_result` is the root cause result; `confidence_score` is the confidence score; `risk_level` is the risk level; `suggested_actions` are the suggested actions; `generation_time_ms` is the data generation time; and `token_usage` is the token usage. In some examples, if the large language model output times out or fails, a rule-based matching approach can be used to diagnose the anomaly and determine its root cause. The rule base can be pre-configured with root cause diagnosis rules based on entropy features and anomaly types, and the root cause result can be determined based on the rules matching the entropy features and anomaly types. For example, root cause diagnosis rules in the rule base may include, but are not limited to, "If the compression ratio is greater than 50, the root cause is determined to be an infinite loop or retry storm," and "If the compression ratio is greater than 1.0 and less than 1.2, the root cause is determined to be a character set error or binary data leakage." If rule base matching is used to diagnose anomalies, the confidence level can be set to a preset value, which indicates a low confidence level; for example, the preset value could be 0.6. In some examples, if the anomaly root cause report data output by the large language model contains sensitive information, the sensitive information can be filtered, such as replacing it with security prompts to indicate that specific information can be obtained by viewing the original logs. When generating anomaly root cause report data, alarm information can also be generated to alert on detected anomalies. In some examples, anomaly event data and anomaly root cause report data can be sent to an alarm platform, which can then push them to users so that users can react quickly and improve the security of business system operations.

[0047] In this embodiment, when a new log appears, the entropy feature of the binary data block of the newly appearing log can be obtained. Based on the entropy feature of the binary data block of the log within a historical time period, an anomaly detection benchmark range suitable for the current situation can be determined. By analyzing the relationship between the newly obtained entropy feature and the anomaly detection benchmark range, the anomaly is identified, the corresponding binary data block is locked, and anomaly event data that can characterize the anomaly is generated. Leveraging the reasoning capabilities of a large language model, based on the prompt words constructed from the locked binary data block and the anomaly event data, anomaly root cause report data characterizing the root cause of the anomaly can be obtained. The entropy feature reflects the regularity and information density of the data in the binary data block, without relying on the semantics of the log. Through the entropy feature, anomalies caused by the business system entering an infinite loop, binary data flowing into the log, etc., can be identified, enabling more comprehensive anomaly detection without semantic dependence. This fills the technical blind spot of anomaly detection relying on semantic content parsing, reduces missed anomaly identification, and improves the anomaly identification rate. Through testing, the anomaly detection method based on log entropy features in this embodiment of the application improved the detection rate of anomalies caused by infinite loop repeated screen refresh from 0% to greater than or equal to 99.2%, and the detection rate of anomalies caused by binary garbled characters, data leakage, etc., from 5.3% to greater than or equal to 97.8%. It can be seen that the anomaly detection method based on log entropy features in this embodiment of the application can significantly improve the anomaly recognition rate.

[0048] Moreover, entropy features are lightweight data, and their transmission and processing consume fewer resources, avoiding the problem of "computing power inversion" under high load scenarios. Through testing, using the anomaly detection method based on log entropy features in this embodiment of the application, if the binary data block size is 4KB, the processing latency for a single binary data block is less than or equal to 0.8 milliseconds, the CPU utilization rate is stable at less than or equal to 8%, and the CPU utilization rate does not increase during traffic peaks. The processable log query per second (QPS) can be increased from 10k to greater than or equal to 500k, and the memory usage is less than or equal to 128MB. It can be seen that the anomaly detection method based on log entropy features in this embodiment of the application can significantly reduce computing power and can easily cope with traffic peaks.

[0049] Furthermore, after detecting an anomaly and obtaining the anomaly event data characterizing the anomaly, the reasoning capabilities of a large language model can be utilized to achieve automated closed-loop analysis of the anomaly's root cause, reducing the rate of manual intervention. Through testing, the anomaly detection method based on log entropy features in this embodiment of the application achieves a closed-loop delay of less than or equal to 3.5 seconds from anomaly detection to the output of the anomaly root cause report data. The accuracy of root cause analysis for binary garbled characters is also greater than or equal to 85%, and the rate of manual intervention decreases from 100% to less than or equal to 15%. Therefore, the anomaly detection method based on log entropy features in this embodiment of the application can reduce root cause analysis time, improve root cause analysis efficiency, increase root cause analysis accuracy, and significantly reduce the rate of manual intervention.

[0050] In some embodiments, a robust adaptive anomaly detection benchmark range can be obtained based on a dual-anchoring mechanism of entropy features within historical timeframes and newly obtained entropy features. Specifically, entropy features within the same time bucket period as the timestamp of the new log's binary data block can be obtained from the entropy features of binary data blocks within historical timeframes as historical samples; a static benchmark range is generated based on the quantiles of the historical samples; the initial drift amount corresponding to the sliding window at the current position is obtained based on the exponentially weighted moving average of the sliding window at the previous position, the newly obtained entropy features, and the quantiles of the historical samples; the static benchmark range is corrected based on the initial drift amount to obtain the anomaly detection benchmark range.

[0051] The process of determining the anomaly detection baseline range by the central processing platform can be triggered periodically, for example, by updating the anomaly detection baseline range every 5 minutes; the process of determining the anomaly detection baseline range by the central processing platform can also be triggered when new entropy features are acquired, and there is no limitation on this.

[0052] Entropy features within a historical timeframe can be pre-divided into time buckets, each corresponding to a time bucket period. A time bucket can be created for a preset duration, which can be set based on scenarios, needs, and experience, such as 1 minute, 5 minutes, 15 minutes, or 60 minutes, but is not limited to these. For example, a 24-hour day can be divided into 288 time buckets, with each bucket divided into 5-minute intervals. Each time bucket includes entropy features within its historical time bucket period. A maximum retention limit can be set for each time bucket; for example, a maximum of 1440 entropy features can be retained per time bucket. A time bucket identifier can be set, indicating the time bucket period. For example, the time bucket identifier can be calculated using `bucket_id = (hour * 60 + minute) / 5`, where `bucket_id` is the time bucket identifier, `hour` is the hour in the timestamp, and `minute` is the minute in the timestamp. Based on the timestamp of the newly obtained entropy feature, the corresponding time bucket identifier can be calculated. The target time bucket is then determined within the time bucket corresponding to the historical duration based on the time bucket identifier. The entropy feature within the target time bucket is the entropy feature within the same time bucket period as the event stamp within the historical duration. The entropy feature within the target time bucket is then identified as a historical sample. For example, if the timestamp of a newly appearing log's binary data block indicates 9:00, the time bucket identifier can be 108. The time bucket period of the entropy feature within the time bucket with time bucket identifier 108 is the same as the time bucket period of the timestamp of the newly appearing log's binary data block. The entropy feature within the time bucket with time bucket identifier 108 is then considered a historical sample. For example, if the historical duration is 7 days and the entropy feature is the compression ratio, the resulting historical samples can form a historical sample set, which can be represented as... ,in, For historical sample sets, This represents the historical sample of day i within the previous M days corresponding to the k-th newly obtained entropy feature. For ease of recording, the historical sample set may include not only the historical samples themselves but also the date and timestamp of each sample. For example, if the entropy feature includes a compression ratio, the historical sample set can also be called the HistoricalSamples list. The historical sample set may include multiple data groups, each containing {cr_value, date_str, timestamp}, where cr_value is the compression ratio, date_str is the date, and timestamp is the timestamp. In some examples, a metadata group can also be configured for the historical sample set, which may include the historical sample set status, the last update time, and the number of samples. For example, a metadata group, also known as BaselinMeta, can be represented as {status, last_update_time, sample_count}. Here, `status` represents the status of the historical sample set, which can include, but is not limited to, three states: Ready, ColdStart, and Insufficient. Ready indicates that the quantity and quality of historical samples in the set meet the requirements; ColdStart indicates that the quality of historical samples meets the requirements, but the quantity is insufficient; and Insufficient indicates that the quality of historical samples does not meet the requirements. `last_update_time` is the most recent update time, and `sample_count` is the number of samples. The historical sample set and metadata group can be stored in embedded data, such as in a local SQLite file. The SQLite table structure can include time bucket identifiers and date partitions. In some examples, if the database storing entropy features within a historical timeframe is corrupted, it can degrade to a memory sliding window mode, retrieving entropy features within the most recent preset timeframe, such as the entropy features within the most recent hour, and continuously attempting to repair or rebuild the database.

[0053] After obtaining multiple historical samples, these samples can be sorted to obtain multiple quantiles. These multiple quantiles may include, but are not limited to, the median and two other quantiles. The quantile interval of the other two quantiles can be calculated. Based on the median, quantile interval, and a preset robustness coefficient, a static benchmark range is determined. Specifically, the first product of the quantile interval and the robustness coefficient can be calculated, along with the first difference between the median and the first product, and the first sum of the median and the first product. The static benchmark range is obtained based on the first difference and the first sum, where the first sum is the upper limit of the static benchmark range, and the first difference is the lower limit. For example, the median of historical samples in the historical sample set can be calculated. 25th percentile and the 75th percentile The interquartile range is the interquartile range. Then the static reference range can be obtained according to the following formula (4): (4) in, This is the static reference range; The robustness coefficient ranges from 1.0 to 3.0, with a default value of 1.5. For ease of recording, static baseline data (StaticBaseline) can be created, represented as {median: float, iqr: float, lower_bound: float, upper_bound: float, lambda: float}, where median is the median, iqr is the quantile interval, lower_bound is the lower limit of the static baseline range, upper_bound is the upper limit of the static baseline range, and lambda is the robustness coefficient. The median offers better robustness and is less sensitive to outliers. Compared to the mean or variance, it reduces the false positive rate in log anomaly detection scenarios. In some examples, there may be severely skewed historical samples, such as a quantile interval of 0. In such cases, a default adjustment value can be used, with the difference between the median and the default adjustment value serving as the lower limit of the static baseline range, and the sum of the median and the default adjustment value serving as the upper limit of the static baseline range, to prevent the static baseline range from being too narrow. The default adjustment value can be set according to the scenario, requirements, experience, etc., and is not limited here; for example, the default adjustment value can be 0.5. In some examples, there may be insufficient historical samples. In this case, a cold start approach can be used, where the median can be replaced with the default baseline value, and a default distribution coefficient can be set. For example, when the entropy feature is compression ratio, the default baseline value can be... It can be 3.5, the default distribution coefficient. It can be 1.0.

[0054] The central processing platform can subscribe to real-time entropy features and maintain a sliding window. Each time a new entropy feature is obtained, the sliding window slides once, moving from the previous position to the current position, and incorporating the newly obtained entropy feature into the sliding window. The sliding window can contain multiple entropy features, arranged in the order of their generation. For example, the sliding window can contain 10 entropy features, but is not limited to this. Specifically, the sliding window can be an Exponentially Weighted Moving Average (EWMA) sliding window. The exponentially weighted moving average of the sliding window at the previous position can be calculated based on the exponentially weighted moving average of the sliding window at the previous position and the newly obtained entropy feature; the initial drift amount corresponding to the sliding window at the current position can be obtained based on the exponentially weighted moving average of the sliding window at the current position and the median of the historical samples. This initial drift amount reflects the degree of drift of the exponentially weighted moving average of the sliding window at the current position relative to the median of the historical samples. For example, the entropy feature includes the compression ratio, and the initial drift amount can be obtained according to the following formulas (5) and (6): (5) (6) in, The exponentially weighted moving average of the sliding window at the current position; The smoothing factor corresponds to a half-life of 2.3 sampling periods and can range from 0.1 to 0.9, with a possible value of 0.3. If the service is highly sensitive, the smoothing factor can be further increased, for example, it can be 0.5. The newly obtained compression ratio; It is the exponentially weighted moving average of the sliding window at the previous position; This is the initial drift amount; This represents the median of historical samples. In some examples, if the real-time stream of entropy features is interrupted, and the interruption duration does not exceed the offline duration (e.g., less than 5 minutes), the most recently obtained exponentially weighted moving average of the sliding window can be marked as stale and used as the exponentially weighted moving average of the sliding window at the current position. If the interruption duration exceeds the offline duration, the system enters the offline state, terminating the process of determining the anomaly detection baseline range.

[0055] In some examples, surge determination can be performed on the initial drift amount to identify persistent fluctuations among multiple entropy features. A preset surge threshold can be set to determine the cause of the initial drift amount. The preset surge threshold can be set based on a multiple of the quantile interval, and is not limited here. For example, the value range of the preset surge threshold can be 1.5 to 4.0, and 2.5 can be selected. In some examples, the preset surge threshold can be adaptively adjusted according to the actual situation. If the absolute value of the initial drift amount is greater than the preset surge threshold, it is determined that the initial drift amount is caused by normal expansion or contraction of the service corresponding to the newly obtained log; if the absolute value of the initial drift amount is less than or equal to the preset surge threshold, it is determined that the initial drift amount is caused by normal jitter of the service corresponding to the newly obtained log. If the absolute value of the initial drift is greater than the preset surge threshold, it indicates a large amplitude of continuous fluctuation in the entropy feature, which is likely due to normal scaling up or down of the service corresponding to the log. In this case, the correction of the static baseline range based on the initial drift is equivalent to shifting the static baseline range, including upward and downward shifts. If the absolute value of the initial drift is less than or equal to the preset surge threshold, it indicates a small amplitude of continuous fluctuation in the entropy feature, which is likely due to normal jitter in the service corresponding to the log. In this case, the correction of the static baseline range based on the initial drift is equivalent to micro-correction. In some examples, surge misjudgments may occur. When a user discovers a surge misjudgment, they will send surge misjudgment information to the central processing platform. Upon receiving the surge misjudgment information input by the user, the central processing platform can roll back the anomaly detection baseline range to the anomaly detection baseline range corresponding to the previous sliding window position. That is, it re-adopts the anomaly detection baseline range determined when the new entropy feature was obtained, and can also learn and update the preset surge threshold.

[0056] To suppress excessive drift in the static reference range, a maximum drift limit and a minimum drift limit can be introduced. If the initial drift is greater than or equal to the minimum drift limit and less than or equal to the maximum drift limit, the static reference range can be directly corrected using the initial drift. Specifically, the upper and lower limits of the static reference range can be increased by the initial drift to obtain the anomaly detection reference range. If the initial drift is less than the minimum drift limit, the static reference range is corrected using the minimum drift limit. Specifically, the upper and lower limits of the static reference range can be increased by the minimum drift limit to obtain the anomaly detection reference range. If the initial drift is greater than the maximum drift limit, the static reference range is corrected using the maximum drift limit. Specifically, the upper and lower limits of the static reference range can be increased by the maximum drift limit to obtain the anomaly detection reference range. The maximum and minimum drift limits can be set based on the quantile distance in the above embodiments. For example, entropy features include compression ratio, quantile is interquartile range, and the anomaly detection baseline range can be obtained according to the following equations (7) and (8): (7) (8) in, This is the baseline range for anomaly detection; This is the initial drift amount; This is the lower limit of the static reference range; This represents the upper limit of the static reference range; This is the minimum drift limit; This is the maximum drift limit; The interquartile range is used. By setting the maximum and minimum drift limits, it is possible to prevent outliers in the entropy features from contaminating the static baseline range, thereby improving the reliability of the anomaly detection baseline range.

[0057] To facilitate recording, a dynamic baseline data structure, namely the DynamicBaseline structure, can be created, which includes information such as the anomaly detection baseline range. For example, the dynamic baseline data structure can be in JSON format, and can be represented as follows: json { "timestamp": XXXXX, "median_hist": 4.2, "current_ewma": 6.8, "delta_cr": 2.6, "dynamic_lower": 2.5, "dynamic_upper": 7.1, "iqr": 1.3, "is_surge": true, "surge_reason": "High traffic expansion" } Wherein, timestamp is the timestamp; median_hist is the median of the historical samples; current_ewma is the exponentially weighted moving average of the sliding window at the current position; delta_cr is the initial drift; dynamic_lower is the lower limit of the anomaly detection benchmark range; dynamic_upper is the upper limit of the anomaly detection benchmark range; iqr is the interquartile range corresponding to the historical samples; is_surge is the surge determination result, true indicates that the absolute value of the initial drift is greater than the preset surge threshold; surge_reason is the surge reason, High traffic expansion indicates that the service corresponding to the log has undergone normal expansion.

[0058] In some examples, the entropy feature includes the compression ratio, and the anomaly detection conditions include a first condition and a second condition. The first condition corresponds to a redundancy anomaly, which includes N consecutive newly obtained compression ratios exceeding the upper limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The second condition corresponds to a discrete anomaly, which includes N consecutive newly obtained compression ratios being less than the lower limit of the anomaly detection benchmark range and less than a preset minimum compression ratio threshold. Specific details regarding redundancy and discrete anomalies can be found in the relevant descriptions in the above embodiments, and will not be repeated here. If the N consecutive newly obtained compression ratios exceed the upper limit of the anomaly detection benchmark range, it indicates that the data in the log has extremely high regularity and extremely low information density; that is, the data with repetitive patterns in the log continues to increase, confirming the occurrence of a redundancy anomaly. If the N consecutive newly obtained compression ratios are less than the lower limit of the anomaly detection benchmark range and less than a preset minimum compression ratio threshold, it indicates that the data in the log has extremely low regularity and extremely high information density; that is, the high-entropy randomized data in the log continues to increase, confirming the occurrence of a discrete anomaly. The value of N can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the value of N can range from 1 to 10, and N can be 3. The minimum compression ratio threshold can be the threshold of the compression ratio of the binary data block after compression, which is close to the uncompressed state. If the compression ratio is less than the minimum compression ratio threshold, it means that the binary compressed block is close to the uncompressed state after compression. The minimum compression ratio threshold can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the minimum compression ratio threshold can be 1.2.

[0059] In other examples, the entropy feature includes information entropy, and the anomaly detection conditions include a third and a fourth condition. Under normal binary data block quality, information entropy and compression ratio are negatively correlated. The third condition corresponds to redundancy anomalies, which includes N consecutive newly obtained information entropies being less than the lower limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The fourth condition corresponds to discrete anomalies, which includes N consecutive newly obtained information entropies being greater than the upper limit of the anomaly detection benchmark range and greater than a preset maximum information entropy threshold. If the N consecutive newly obtained information entropies are less than the lower limit of the anomaly detection benchmark range, it indicates that the data in the log has extremely high regularity and extremely low information density; that is, the data with repetitive patterns in the log continues to increase, confirming the occurrence of a redundancy anomaly. If the N consecutive newly obtained information entropies are greater than the upper limit of the anomaly detection benchmark range and greater than a preset maximum information entropy threshold, it indicates that the data in the log has extremely low regularity and extremely high information density; that is, the high-entropy randomized data in the log continues to increase, confirming the occurrence of a discrete anomaly. The value of N can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the value of N can range from 1 to 10, and N can be 3. The maximum information entropy threshold can be the threshold of information entropy of the binary data block after compression that is close to an uncompressed state. If the information entropy is greater than the maximum information entropy threshold, it means that the binary compressed block is close to an uncompressed state after compression. The maximum information entropy threshold can be set according to the scenario, requirements, experience, etc., and is not limited here.

[0060] The anomaly detection in this embodiment is continuous. It maintains the 10 most recently obtained sliding windows and detects anomalies by observing the consistency of the direction in which newly obtained entropy features cross the anomaly detection benchmark range, thus determining the anomaly type. Continuous detection avoids false positives caused by business fluctuations. If a newly obtained entropy feature falls within the anomaly detection benchmark range, the counters recording the number of times the entropy feature is less than the lower limit and the number of times it is greater than the upper limit are reset to zero. Through testing, the anomaly detection method based on log entropy features in this embodiment, through continuous detection and dynamic correction of the anomaly detection benchmark range, achieves an anomaly false positive rate of less than or equal to 2%, which is significantly lower than the anomaly false positive rate of static threshold anomaly detection schemes.

[0061] In some embodiments, anomaly detection can be performed using an isolated forest model combined with an anomaly detection benchmark range. The central processing platform can input entropy features from the historical time period and newly obtained entropy features into a pre-trained isolated forest model to obtain an anomaly score output by the isolated forest model; based on the average value of the entropy features over the historical time period, an anomaly detection benchmark range is determined. The isolated forest model can be pre-trained. An isolated forest model is an unsupervised learning model and can include multiple isolated trees. A portion of the data can be randomly sampled to train a single isolated tree; for example, an isolated forest model can include 100 isolated trees, with a sampling ratio of 0.256. To improve the accuracy of the isolated forest model's output, updated entropy features can be periodically used as samples to train the isolated forest model, achieving dynamic updates. Using an isolated forest model, it is not necessary to bin the entropy features over the historical time period by time. The isolated forest model can be used to output an anomaly score for newly obtained entropy features based on the time series of the input entropy features. This anomaly score characterizes the degree of deviation of the newly obtained entropy features from the entropy features over the historical time period. The anomaly score is positively correlated with the degree of deviation; the higher the degree of deviation, the higher the anomaly score, and the more likely an anomaly is to occur. The anomaly detection benchmark range can be determined based on the average value of entropy features over a historical period. This average value can be used as the baseline for defining the anomaly detection benchmark range. The anomaly detection benchmark range includes a first anomaly detection benchmark range and a second anomaly detection benchmark range. The first anomaly detection benchmark range is the range greater than the average value of entropy features over a historical period, and the second anomaly detection benchmark range is the range less than the average value of entropy features over a historical period.

[0062] In some examples, when the entropy feature includes compression ratio, the anomaly detection conditions include a fifth and a sixth condition. The fifth condition corresponds to redundant anomalies, and includes the candidate compression ratio being within the first anomaly detection benchmark range. The candidate compression ratio includes newly obtained compression ratios with anomaly scores greater than a preset score threshold. The sixth condition corresponds to discrete anomalies, and includes the candidate compression ratio being within the second anomaly detection benchmark range. The preset score threshold is the threshold value for determining the degree of deviation to an anomaly. It can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, if the anomaly score range is 0 to 1, the preset score threshold can be set to 0.7. An anomaly score greater than the preset score threshold indicates that the business corresponding to the newly obtained compression ratio has an anomaly. The anomaly type can be further determined by comparing the newly obtained compression ratio with the average entropy feature over a historical period. If the newly obtained compression ratio is greater than the average entropy feature over a historical period, i.e., the newly obtained compression ratio is within the first anomaly detection benchmark range, it indicates that the data in the log has extremely high regularity and extremely low information density. That is, the data with repetitive patterns in the log continues to increase, and a redundant anomaly can be identified. If the newly obtained compression ratio is less than the average value of the entropy features within the historical time period, that is, the newly obtained compression ratio is within the range of the second anomaly detection benchmark, it indicates that the regularity of the data in the log is extremely low and the information density is extremely high. In other words, the high-entropy randomized data in the log continues to increase, and it can be determined that a discrete anomaly has occurred.

[0063] In other examples, where the entropy feature includes information entropy, the anomaly detection conditions include a seventh and an eighth condition. The seventh condition corresponds to redundant anomalies, and includes the anomaly candidate information entropy being within the second anomaly detection benchmark range. The anomaly candidate information entropy includes newly obtained information entropy with an anomaly score greater than a preset score threshold. The eighth condition corresponds to discrete anomalies, and includes the anomaly candidate information entropy being within the first anomaly detection benchmark range. For details on anomaly scores and preset score thresholds, please refer to the relevant explanations in the examples above, which will not be repeated here. If the newly obtained information entropy is less than the average value of the entropy feature over the historical time period, i.e., the newly obtained information entropy is within the second anomaly detection benchmark range, it indicates that the data in the log has extremely high regularity and extremely low information density; that is, the data with repetitive patterns in the log continues to increase, and a redundant anomaly can be identified. If the newly obtained information entropy is greater than the average value of the entropy feature over the historical time period, i.e., the newly obtained information entropy is within the first anomaly detection benchmark range, it indicates that the data in the log has extremely low regularity and extremely high information density; that is, the high-entropy randomized data in the log continues to increase, and a discrete anomaly can be identified.

[0064] Anomaly detection is performed by combining the isolated forest model with an anomaly detection benchmark range. This method is suitable for scenarios without obvious business cycles, long-term operating systems with complex load distribution and sufficient historical data to meet model training requirements.

[0065] In some embodiments, after obtaining the root cause report data, the root cause report data and the abnormal event data can be merged and pushed to the alarm platform so that the alarm platform can provide the user with corresponding notification or alarm information. The central processing platform can push data to the alarm platform via Webhook, but is not limited to this. The root cause report data is associated with the business process identifier. Specifically, the root cause report data may include an abnormal event identifier. The corresponding abnormal event data can be located based on the abnormal event identifier. The abnormal event data may include the data block identifier of a locked binary data block. The corresponding entropy feature can be located based on the data block identifier of the locked binary data block. The entropy feature may include the business process identifier. That is, the root cause report data and the business process identifier are associated. If two or more of the aforementioned root cause report data associated with the same business process identifier are obtained within a preset aggregation time, the two or more of the aforementioned root cause report data and the corresponding abnormal event data are merged into an abnormal event alarm message and pushed to the alarm platform. The preset aggregation time can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the preset aggregation time can be 10 seconds. Multiple root cause reports associated with the same task process identifier within a preset aggregation period, along with corresponding abnormal event data, can be merged into a single abnormal event alarm message and pushed to the alarm platform to avoid alarm storms in a short period. In some examples, the number of root cause reports that can be merged within the preset aggregation period can also be limited; for example, a single abnormal event alarm message can merge a maximum of five root cause reports corresponding to different abnormalities.

[0066] For ease of understanding, the following uses entropy features, including compression ratio, as an example to illustrate the anomaly detection method based on log entropy features in the embodiments of this application. Figure 3 A schematic diagram illustrating an example of an anomaly detection method based on log entropy features provided in this application embodiment, as shown below. Figure 3 As shown, the anomaly detection system is the log entropy-based anomaly detection system in this application embodiment, and the business process is the process of the business executed by the business system. The log entropy-based anomaly detection method may include steps a1 to a13.

[0067] In step a1, the anomaly detection system captures the log writing of the business process.

[0068] In step a2, the anomaly detection system fragments the binary data of the log to obtain binary data blocks.

[0069] In step a3, the anomaly detection system uses a compression algorithm to compress binary data blocks and calculates the compression ratio.

[0070] In step a4, the anomaly detection system determines the anomaly detection baseline range.

[0071] In step a5, the anomaly detection system compares the newly obtained compression ratio with the anomaly detection benchmark range. If the newly obtained compression ratio is greater than the upper limit of the anomaly detection benchmark range for N consecutive times, step a6 is executed; if the newly obtained compression ratio is less than the lower limit of the anomaly detection benchmark range for N consecutive times and is less than the minimum compression ratio threshold, step a10 is executed.

[0072] In step a6, the anomaly detection system triggers an infinite loop / screen refresh alarm to the alarm platform.

[0073] In step a7, the anomaly detection system uploads a prompt word request for analysis to the large language model. The prompt word includes the raw data of the locked binary data block.

[0074] In step a8, the large language model system returns anomaly root cause analysis report data to the anomaly detection system.

[0075] In step a9, the anomaly detection system sends anomaly root cause analysis report data to the alarm platform.

[0076] Steps a6 to a9 above constitute the process for handling redundant anomalies.

[0077] In step a10, the anomaly detection system triggers a garbled character / data leakage alarm to the alarm platform.

[0078] In step a11, the anomaly detection system uploads a prompt word request for analysis to the large language model. The prompt word includes the raw data of the locked binary data block.

[0079] In step a12, the large language model system returns anomaly root cause analysis report data to the anomaly detection system.

[0080] In step a13, the anomaly detection system sends anomaly root cause analysis report data to the alarm platform.

[0081] Steps a10 to a13 above constitute the process for handling discrete anomalies.

[0082] The specific details of steps a1 to a13 above can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0083] In some embodiments, the functionality of the anomaly detection system based on log entropy features in the above embodiments can be implemented through a streaming sampling module, an entropy feature generation engine, a dynamic baseline manager, a bidirectional anomaly determiner, and a root cause analysis agent. Figure 4 A schematic diagram illustrating an example of an anomaly detection system based on log entropy features provided in this application embodiment, as shown below. Figure 4As shown, the edge node device 11 may include a streaming sampling module 111 and an entropy feature generation engine 112, and the central processing platform 12 may include a dynamic baseline manager 121, a bidirectional anomaly determiner 122, a root cause analysis agent 123, and a data receiving gateway 124.

[0084] When the business process 13 writes to the log file 14, the streaming sampling module 111 can acquire the binary data of the newly written log and segment the continuous binary data into fixed-size binary data blocks. These binary data blocks are buffered in a local lock-free circular buffer. After successfully transmitting a binary data block to the entropy feature generation engine 112, the successfully transmitted binary data block is deleted from the lock-free circular buffer. The streaming sampling module 111 can determine whether it receives an acknowledgment (ACK) from the entropy feature generation engine 112. If the entropy feature generation engine 112 successfully receives and consumes a binary data block, it sends an ACK to the streaming sampling module 111, and the streaming sampling module 111 deletes this binary data block from the lock-free circular buffer. If the streaming sampling module 111 does not receive an ACK within a timeout, it is considered a transmission failure, and information can be recorded in the log of the anomaly detection system, and binary data sampling can continue. To facilitate the transmission of binary data blocks to the entropy feature generation engine 112, the binary data blocks can be serialized. The serialization format can be Protobuf, but is not limited to it. The encoding time for serialization is less than 0.1 milliseconds, which is negligible. The streaming sampling module 111 can send the serialized binary data blocks to the entropy feature generation engine 112 through ZeroMQ PUSH-PULL mode. The communication between the streaming sampling module 111 and the entropy feature generation engine 112 is inter-process communication (IPC), and the latency can be less than 0.2 milliseconds. The compression method of ZeroMQ information frames can be Snappy, which can reduce IPC communication overhead by about 30%. In some examples, a high water mark (HWM) can be configured for the sender in ZeroMQ PUSH-PULL mode to prevent memory bloat. The high water mark can be set according to the scenario, requirements, experience, etc., and is not limited here. For example, the high water mark can be set to 50. In some examples, local disk buffering (spill todisk) can be enabled when the ZeroMQ connection is lost. A single file on the local disk can be up to 50MB, and the 10 most recent files can be retained. The streaming sampling module 111 has a distribution throughput of greater than or equal to 200MB / s, and an end-to-end binary data sampling latency of less than or equal to 2 milliseconds.

[0085] The entropy feature generation engine 112 can obtain serialized binary data blocks from ZeroMQ message frames, deserialize them to obtain the original data of the binary data blocks, and obtain the entropy features of the binary data blocks based on the original data. The entropy features can be stored in a local time-series database and can be transmitted to the dynamic baseline manager 121, for example, by streaming the entropy features to the dynamic baseline manager 121 via gRPC. When the entropy features include a compression ratio, the entropy feature generation engine 112 can also perform cross-validation of the binary data block's compression ratio and information entropy to obtain a quality label for the binary data block. The entropy feature generation engine 112 can output the entropy features with the quality label to the data receiving gateway 124 in the central processing platform 12.

[0086] The dynamic baseline manager 121 acquires entropy features from the data receiving gateway 124. It can retrieve entropy features from the local time-series database that have timestamps within the same time bucket period as the newly acquired entropy features as historical samples. The retrieval latency in the time-series database is less than or equal to 50 milliseconds, and the database size can be less than 500 bytes, accommodating approximately 30 days of entropy features. A static baseline range is generated based on the historical samples. The initial drift is obtained in real-time based on the exponentially weighted moving average of the sliding window, the newly acquired entropy features, and the historical samples. The static baseline range is then corrected based on the initial drift to obtain the anomaly detection baseline range. The latest anomaly detection baseline range can be cached in memory, and historical changes to the anomaly detection baseline range can be stored in a local SQLite database for use when backtracking or review is needed. The update latency of the exponentially weighted moving average of the sliding window is less than or equal to 0.1 milliseconds, the update cycle of the anomaly detection baseline range can be 5 minutes, and the memory usage of the sliding window and cache can be less than 32 MB.

[0087] The bidirectional anomaly determiner 122 can obtain real-time entropy features from the entropy feature generation engine 112 and anomaly detection baseline range from the dynamic baseline manager 121. Based on the relationship between the newly obtained entropy features and the anomaly detection baseline range, the bidirectional anomaly determiner 122 can identify anomalies and their types, and generate anomaly event data. The bidirectional anomaly determiner 122 can communicate via the Hypertext Transfer Protocol (HTTP) POST method. In some examples, the same data block identifier triggers only one generation of anomaly event data within a repetition suppression duration to prevent duplicate alarms. The repetition suppression duration can be set according to the scenario, requirements, experience, etc.; for example, the repetition suppression duration can be 30 seconds. The bidirectional anomaly determiner 122 can send real-time anomaly event data to the alarm center 15.

[0088] The root cause analysis agent 123 obtains anomalous event data from the bidirectional anomaly determiner 122. It can extract the original data and anomaly type of the locked binary data block from the anomalous events to construct prompt words. Based on these prompt words, it calls the large language model 16 to obtain anomaly root cause report data. The choice of the large language model is not limited here; an existing mature large language model or a large language model pre-trained based on the business domain can be selected. A temperature parameter for the large language model can be set, ranging from 0.1 to 1.0. To ensure the determinism of root cause analysis, a lower temperature parameter can be set, for example, to 0.3. To control the output token quantity of the large language model, a maximum output token quantity can be set, for example, 2048, but not limited to this. To balance the information content and the token cost of the large language model, a portion of the original data of the binary data block can be truncated and added to the prompt words. The length of the truncated bytes can be between 256 and 1024 bytes, for example, 512 bytes, but not limited to this. After constructing the prompt word, an exception request data structure, AnalysisRequest, can be generated. The format of the exception request data can be {request_id, prompt, model_config, timeout}, where request_id is the exception request identifier, prompt is the prompt word, model_config is the model parameters, and timeout is the timeout period. In some examples, if the exception is a discrete exception, the root cause analysis agent 123 can also provide byte frequency analysis auxiliary data for binary data blocks. The format of the byte frequency analysis auxiliary data can be {top_bytes: [...], entropy_calculated: 7.8}, where top_bytes are the most frequent bytes and their frequency, and entropy_calculated is the information entropy of the binary data block. The prompt word construction delay is less than 10 milliseconds and can be achieved through local string manipulation. A CRC check can be performed on the original data of the binary data block, i.e., the Hex data. If the check fails, a request is sent to the streaming sampling module 111 to retransmit the binary data block. In this embodiment, the entire process from the occurrence of an anomaly to its detection and the generation of root cause report data takes less than or equal to 5 seconds, demonstrating fast and efficient anomaly detection. The root cause analysis agent 123 can communicate with the alarm platform 15 via WebhookPOST to send root cause report data to the alarm platform 15.

[0089] In the steps executed by the streaming sampling module 111, entropy feature generation engine 112, dynamic baseline manager 121, bidirectional anomaly determiner 122, and root cause analysis agent 123, if a failure occurs, an exponential backoff approach can be adopted for retrying. A maximum number of retries can be set; if the maximum number of retries is reached, no further retries will be made, for example, a maximum of 3 retries. The streaming sampling module 111 and entropy feature generation engine 112 can adopt a breakpoint resumption mechanism. If a process restart occurs, the subsequent process can continue from the last sent and received ACK position. In the above process, the metadata header, entropy features, anomaly event data, and anomaly root cause report data of the binary data block can carry unique identifiers, such as data block identifiers and anomaly event identifiers. The receiving end can use an identifier deduplication table to ensure idempotent processing. In this embodiment, when the downstream module is unavailable, a local file buffer can be enabled, allowing the anomaly detection system to maintain "degraded operation" rather than "complete failure," thus ensuring the reliability of anomaly detection while minimizing anomaly detection interruptions.

[0090] This application also provides an anomaly detection device based on log entropy features, which corresponds to the anomaly detection device based on log entropy features in the above embodiments and can be applied to a central processing platform. Figure 5 This is a schematic diagram of the structure of an anomaly detection device based on log entropy features provided in an embodiment of this application, as shown below. Figure 5 As shown, the anomaly detection device 300 based on log entropy features may include an entropy feature acquisition module 301, a baseline determination module 302, an anomaly determination module 303, and a root cause analysis module 304.

[0091] The entropy feature acquisition module 301 can be used to acquire newly obtained entropy features, which are the entropy features of newly appearing binary data blocks of logs.

[0092] The baseline determination module 302 can be used to determine the anomaly detection baseline range based on the entropy characteristics of binary data blocks in logs over a historical period.

[0093] The anomaly determination module 303 can be used to lock the binary data block corresponding to the newly obtained entropy feature and generate anomaly event data when the relationship between the newly obtained entropy feature and the anomaly detection benchmark range meets the anomaly detection conditions.

[0094] The root cause analysis module 304 can be used to construct prompt words based on locked binary data blocks and abnormal event data, and call a large language model to infer and output abnormal root cause report data based on the prompt words.

[0095] In some embodiments, newly emerging log binary data blocks are obtained based on binary data fragments of newly emerging logs. Each binary data block has a metadata header, which includes a data block identifier, a timestamp, and a business process identifier. Entropy features include a compression ratio obtained by compressing the binary data block using a preset compression algorithm or information entropy obtained by calculating the binary data block using an entropy algorithm. When the entropy feature includes a compression ratio, the binary data block has a quality label, which is obtained based on the correlation between the compression ratio and information entropy of a subset of binary data blocks. These subsets of binary data blocks are extracted from multiple binary data blocks of the newly emerging log.

[0096] In some embodiments, the benchmark determination module 302 can be used to: obtain entropy features from the entropy features of binary data blocks within a historical time period that are within the same time bucket period as the timestamp of the new log binary data block as historical samples; generate a static benchmark range based on the quantiles of the historical samples; obtain the initial drift amount corresponding to the sliding window at the current position according to the exponentially weighted moving average of the sliding window at the previous position, the newly obtained entropy features, and the quantiles of the historical samples; and correct the static benchmark range according to the initial drift amount to obtain the anomaly detection benchmark range.

[0097] In some examples, if the absolute value of the initial drift is greater than the preset surge threshold, it is determined that the initial drift is caused by the normal expansion or contraction of the service corresponding to the newly obtained log; if the absolute value of the initial drift is less than or equal to the preset surge threshold, it is determined that the initial drift is caused by the normal jitter of the service corresponding to the newly obtained log; if a surge misjudgment information is received from the user, the anomaly detection benchmark range is rolled back to the anomaly detection benchmark range corresponding to the previous sliding window.

[0098] In some examples, the entropy feature includes the compression ratio, and the anomaly detection conditions include a first condition and a second condition. The first condition corresponds to a redundant anomaly, which indicates an increase in repetitive pattern data in the log. The first condition includes N consecutive newly obtained compression ratios that are greater than the upper limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The second condition corresponds to a discrete anomaly, which indicates an increase in high-entropy randomized data in the log. The second condition includes N consecutive newly obtained compression ratios that are less than the lower limit of the anomaly detection benchmark range and less than a preset minimum compression ratio threshold.

[0099] In some examples, the entropy feature includes information entropy, and the anomaly detection conditions include a third condition and a fourth condition. The third condition corresponds to redundant anomalies, which indicate an increase in repetitive patterns of data in the log. The third condition includes N consecutive newly obtained information entropies being less than the lower limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The fourth condition corresponds to discrete anomalies, which indicate an increase in high-entropy randomized data in the log. The fourth condition includes N consecutive newly obtained information entropies being greater than the upper limit of the anomaly detection benchmark range and greater than a preset maximum information entropy threshold.

[0100] In some embodiments, the benchmark determination module 302 is further configured to: input the entropy features from the historical time period and the newly obtained entropy features into the pre-trained isolated forest model to obtain the anomaly score output by the isolated forest model. The benchmark determination module 302 can be used to: determine the anomaly detection benchmark range based on the average value of the entropy features from the historical time period.

[0101] In some examples, the anomaly detection benchmark range includes a first anomaly detection benchmark range and a second anomaly detection benchmark range. The first anomaly detection benchmark range is the range that is greater than the average value of the entropy features over a historical period, and the second anomaly detection benchmark range is the range that is less than the average value of the entropy features over a historical period.

[0102] When the entropy feature includes compression ratio, the anomaly detection conditions include a fifth condition and a sixth condition. The fifth condition corresponds to redundant anomalies, which indicate an increase in repetitive patterns in the log. The fifth condition includes an anomaly candidate compression ratio being within the first anomaly detection benchmark range, and the anomaly candidate compression ratio includes newly obtained compression ratios with anomaly scores greater than a preset score threshold. The sixth condition corresponds to discrete anomalies, which indicate an increase in high-entropy randomized data in the log. The fifth condition includes an anomaly candidate compression ratio being within the second anomaly detection benchmark range.

[0103] When the entropy feature includes information entropy, the anomaly detection conditions include a seventh condition and an eighth condition; the seventh condition corresponds to redundant anomalies, and the seventh condition includes that the anomaly candidate information entropy is within the second anomaly detection benchmark range, and the anomaly candidate information entropy includes newly obtained information entropy with an anomaly score greater than a preset score threshold; the eighth condition corresponds to discrete anomalies, and the eighth condition includes that the anomaly candidate information entropy is within the first anomaly detection benchmark range.

[0104] In some embodiments, the root cause report data is associated with the business process identifier. The root cause analysis module 304 can also be used to merge two or more root cause report data associated with the same business process identifier into one abnormal event alarm message and push it to the alarm platform when two or more root cause report data are obtained within a preset aggregation time.

[0105] It should be noted that the anomaly detection device 300 based on log entropy features is a device corresponding to the anomaly detection method based on log entropy features described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0106] This application also provides an electronic device corresponding to the anomaly detection method based on log entropy features in the above embodiments. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 400 includes a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402.

[0107] In some examples, the processor 402 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0108] Memory 401 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the anomaly detection method based on log entropy features according to embodiments of this application.

[0109] The processor 402 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 401, so as to implement the anomaly detection method based on log entropy features in the above embodiments.

[0110] In some examples, electronic device 400 may also include a communication interface 403 and a bus 404. For example, Figure 6 As shown, the memory 401, processor 402, and communication interface 403 are connected through bus 404 and complete communication with each other.

[0111] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 403.

[0112] Bus 404 includes hardware, software, or both, that couples components of electronic device 400 together. For example, and not as a limitation, bus 404 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0113] This application also provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are executed by a processor, they can implement the anomaly detection method based on log entropy features in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0114] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the anomaly detection method based on log entropy features in the above embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0115] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, system embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0116] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. An anomaly detection method based on log entropy features, characterized in that, include: Obtain the newly obtained entropy feature, which is the entropy feature of the binary data block of the newly appearing log; Based on the entropy characteristics of binary data blocks in historical logs, the baseline range for anomaly detection is determined. If the relationship between the newly obtained entropy feature and the anomaly detection benchmark range satisfies the anomaly detection condition, the binary data block corresponding to the newly obtained entropy feature is locked and anomaly event data is generated. Based on the locked binary data block and the abnormal event data, a prompt word is constructed, and a large language model is invoked to infer and output abnormal root cause report data based on the prompt word.

2. The method according to claim 1, characterized in that, The binary data blocks of newly emerging logs are obtained by fragmenting the binary data of newly emerging logs. The binary data blocks have a metadata header, which includes a data block identifier, a timestamp, and a business process identifier. The entropy feature includes the compression ratio obtained by compressing binary data blocks based on a preset compression algorithm or the information entropy obtained by operating on binary data blocks based on an entropy algorithm. When the entropy feature includes the compression ratio, the binary data block has a quality label, which is obtained based on the correlation between the compression ratio and information entropy of a portion of the binary data block. The portion of the binary data block is extracted from multiple binary data blocks of newly emerging logs.

3. The method according to claim 1, characterized in that, The entropy characteristics of binary data blocks in historical logs are used to determine the anomaly detection baseline range, including: From the entropy features of binary data blocks within a historical time period, the entropy features that are within the same time bucket period as the timestamp of the binary data block of the new log are obtained as historical samples. Based on the quantiles of the historical samples, a static benchmark range is generated; Based on the exponentially weighted moving average of the sliding window at the previous position, the newly obtained entropy feature, and the quantiles of the historical samples, the initial drift amount corresponding to the sliding window at the current position is obtained. The static reference range is corrected based on the initial drift amount to obtain the anomaly detection reference range.

4. The method according to claim 3, characterized in that, If the absolute value of the initial drift is greater than the preset surge threshold, it is determined that the initial drift is caused by the normal expansion or contraction of the service corresponding to the newly obtained log. If the absolute value of the initial drift is less than or equal to a preset surge threshold, it is determined that the initial drift is caused by normal jitter of the service corresponding to the newly obtained log. Upon receiving a surge misjudgment information input by the user, the anomaly detection benchmark range is rolled back to the anomaly detection benchmark range corresponding to the previous position of the sliding window.

5. The method according to claim 3, characterized in that, Entropy features include compression ratio, and the anomaly detection conditions include a first condition and a second condition; The first condition corresponds to a redundant anomaly, which indicates an increase in repetitive pattern data in the log. The first condition includes N consecutive newly obtained compression ratios that are greater than the upper limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The second condition corresponds to a discrete anomaly, which indicates an increase in high-entropy randomized data in the log. The second condition includes N consecutive newly obtained compression ratios that are less than the lower limit of the anomaly detection benchmark range and less than a preset minimum compression ratio threshold.

6. The method according to claim 3, characterized in that, Entropy features include information entropy, and the anomaly detection conditions include a third condition and a fourth condition; The third condition corresponds to a redundant anomaly, which indicates an increase in repetitive pattern data in the log. The third condition includes N consecutive newly obtained information entropy values ​​being less than the lower limit of the anomaly detection benchmark range, where N is a positive integer greater than 1. The fourth condition corresponds to a discrete anomaly, which indicates an increase in high-entropy randomized data in the log. The fourth condition includes N consecutive newly obtained information entropy values ​​that are greater than the upper limit of the anomaly detection benchmark range and greater than the preset maximum information entropy threshold.

7. The method according to claim 1, characterized in that, After obtaining the newly acquired entropy features, the process also includes: The entropy features within the historical time period and the newly obtained entropy features are input into the pre-trained isolated forest model to obtain the anomaly score output by the isolated forest model. The entropy characteristics of binary data blocks in historical logs are used to determine the anomaly detection baseline range, including: The anomaly detection benchmark range is determined based on the average value of the entropy features over the historical time period.

8. The method according to claim 7, characterized in that, The anomaly detection benchmark range includes a first anomaly detection benchmark range and a second anomaly detection benchmark range. The first anomaly detection benchmark range is the range that is greater than the average value of the entropy features within the historical time period, and the second anomaly detection benchmark range is the range that is less than the average value of the entropy features within the historical time period. When the entropy feature includes the compression ratio, the anomaly detection conditions include the fifth condition and the sixth condition; The fifth condition corresponds to a redundant anomaly, which indicates an increase in repetitive pattern data in the log. The fifth condition includes an anomaly candidate compression ratio being within the first anomaly detection benchmark range, and the anomaly candidate compression ratio includes a newly obtained compression ratio where the anomaly score is greater than a preset score threshold. The sixth condition corresponds to a discrete anomaly, which indicates an increase in high-entropy randomized data in the log. The fifth condition includes an anomaly candidate compression ratio being within the second anomaly detection benchmark range. When the entropy feature includes information entropy, the anomaly detection conditions include a seventh condition and an eighth condition; the seventh condition corresponds to the redundant anomaly, and the seventh condition includes that the anomaly candidate information entropy is located within the second anomaly detection benchmark range, and the anomaly candidate information entropy includes the newly obtained information entropy where the anomaly score is greater than a preset score threshold; the eighth condition corresponds to the discrete anomaly, and the eighth condition includes that the anomaly candidate information entropy is located within the first anomaly detection benchmark range.

9. The method according to claim 1, characterized in that, The anomaly root cause report data is associated with the business process identifier; The method further includes: If two or more of the aforementioned root cause reports of anomalies are obtained within a preset aggregation period and are associated with the same business process identifier, the two or more of the aforementioned root cause reports of anomalies and the corresponding anomaly event data are merged into an anomaly event alarm message and pushed to the alarm platform.

10. An anomaly detection device based on log entropy features, characterized in that, include: The entropy feature acquisition module is used to acquire newly obtained entropy features, which are the entropy features of newly appearing binary data blocks in the log. The baseline determination module is used to determine the anomaly detection baseline range based on the entropy characteristics of binary data blocks in the logs over a historical period. The anomaly determination module is used to lock the binary data block corresponding to the newly obtained entropy feature and generate anomaly event data when the relationship between the newly obtained entropy feature and the anomaly detection benchmark range meets the anomaly detection conditions. The root cause analysis module is used to construct prompt words based on the locked binary data block and the abnormal event data, and call the large language model to infer and output abnormal root cause report data based on the prompt words.

11. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the anomaly detection method based on log entropy features as described in any one of claims 1 to 9.

12. An anomaly detection system based on log entropy features, characterized in that, include: Edge node devices are used to respond to the emergence of new logs, obtain the binary data blocks of the newly obtained logs and the entropy features of the binary data blocks, and upload the newly obtained entropy features to the central processing platform. The central processing platform is communicatively connected to the edge node devices and is used to execute the anomaly detection method based on log entropy features as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the anomaly detection method based on log entropy features as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the anomaly detection method based on log entropy features as described in any one of claims 1 to 9.

Images