Web server intrusion detection method

Abstract

The invention relates to the technical field of network security, in particular to a Web server intrusion detection method, which comprises the following steps of: dividing a web request in an audit log into a plurality of request units by utilizing a unit division technology; carrying out API classification on the plurality of request units by utilizing an API classification engine based on a prefix tree algorithm to obtain a plurality of APIs, wherein each API maintains a detection window containing the plurality of request units; calculating feature similarity among the plurality of request units in each detection window, and identifying whether abnormal request units deviating from group behaviors exist in the detection windows or not according to the feature similarity; if the abnormal request unit deviating from the group behavior exists, it is judged that Web server intrusion is detected, and the abnormal request unit is removed. Therefore, the problems that an existing web application anomaly detection technology is limited in application scene, rough in detection granularity, lack of self-adaptive comparison reference and prone to collaborative poisoning attack are solved.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a method for detecting web server intrusion. Background Technology

[0002] Anomaly detection is crucial for web application security, but existing methods such as rule-based validation and learning-based models face significant limitations. Rule-based systems struggle to cope with novel attacks, while learning-based methods require frequent and costly retraining to adapt to dynamic application updates, often resulting in high false positive rates. Although recent self-comparison methods address the retraining problem by comparing replicas in microservice scenarios, they are unsuitable for monolithic applications due to functional heterogeneity and suffer from issues such as coarse detection granularity, lack of adaptive comparison baselines, and vulnerability to coordinated poisoning attacks. Summary of the Invention

[0003] This application provides a web server intrusion detection method to address the problems of existing web application anomaly detection technologies, such as limited applicability, coarse detection granularity, lack of adaptive comparison benchmarks, and susceptibility to collaborative poisoning attacks.

[0004] The first aspect of this application provides a web server intrusion detection method, comprising the following steps: dividing web requests in audit logs into multiple request units using unit partitioning technology; using an API classification engine, identifying the API affiliation of each request unit in real time based on a prefix tree algorithm, and classifying the multiple request units into multiple APIs according to the identification results, wherein each API maintains a detection window containing multiple request units; calculating the feature similarity between multiple request units within each detection window, and identifying whether there are abnormal request units deviating from the group behavior in the detection window based on the feature similarity; if there are abnormal request units deviating from the group behavior, determining that a web server intrusion has been detected, and removing the abnormal request units.

[0005] Optionally, the step of calculating the feature similarity among multiple request units within each detection window and identifying whether there are abnormal request units deviating from the group behavior in the detection window based on the feature similarity includes: calculating the set feature similarity among request units using Jaccard similarity and calculating the sequence feature similarity among request units using cosine similarity; calculating the initial abnormal score of each request unit on each feature dimension based on the set feature similarity and sequence feature similarity; collecting the initial abnormal scores of all request units on each feature dimension to form a vector, and normalizing the vector to obtain a normalized abnormal feature vector; calculating the Euclidean distance between the normalized abnormal feature vector and the origin to obtain the final abnormal score of each request unit; if there is a target request unit whose final abnormal score is greater than a preset value, then the target request unit is determined to be an abnormal request unit.

[0006] Optionally, the web requests in the audit log are divided into multiple request units using unit partitioning technology, including: for multi-process concurrency models or multi-thread concurrency models, using the process ID and thread ID recorded in the audit log, the log entries are grouped by source, and a first delimiter log is inserted before each worker process / thread starts processing a new request, so as to determine the request unit to which the log entry belongs based on the first delimiter log; for coroutine concurrency models, a second delimiter log is inserted each time the web application switches coroutines, so as to determine the request unit to which the log entry belongs based on the second delimiter log.

[0007] Optionally, the step of using an API classification engine to identify the API affiliation of each request unit in real time based on a prefix tree algorithm, and classifying the multiple request units into multiple APIs according to the identification results, includes: constructing at least one API template using a prefix tree structure; using the HTTP request method and URL path of the request unit as the search key, when searching for a matching API template in the prefix tree, determining whether the search key matches the complete path in the prefix tree; if the search key completely matches the complete path in the prefix tree, and the leaf node of the complete path represents a defined API template, then the current request unit is classified as the defined API template; if there is a search key that does not completely match the corresponding API template, then the search key is recorded as a static key for parameterized template generalization.

[0008] Optionally, the step of treating the search key as a static key record and performing parameterized template generalization includes: finding similar key pairs from the static key set that have the same HTTP method and the same number of URL segments as the search key, and whose URL segments all conform to the same predefined parameter type; using the API template corresponding to the similar key pair as the API template of the search key, adding the API template of the search key to the prefix tree, and removing the search key covered by the generalized template.

[0009] Optionally, the above-described web server intrusion detection method further includes: recording the parameter values ​​and timestamps of the parameterized URL segments of the API template;

[0010] When the duration for which the number of parameter value types of any parameterized URL segment is less than a preset threshold is greater than a preset time, a new API path is generated for the parameterized URL segment and added to the prefix tree, and the API template of the parameterized URL segment is removed or modified.

[0011] Optionally, the final anomaly score is:

[0012]

[0013] Among them, D i For requesting unit u i Final abnormal score, Let x' be the normalized anomalous feature vector. ki M represents the initial anomaly score of the normalized anomaly feature vector, where M and k are both the number of features.

[0014] Optionally, the above-described web server intrusion detection method further includes: obtaining the latest request unit that recently entered the API in the detection window maintained by each API; replacing the oldest unit in the detection window with the latest request unit to form a new detection window; if the latest request unit is a normal request unit, then keeping the latest request unit in the new detection window and updating the API directory; if the latest request unit is an abnormal request unit, then discarding the latest request unit.

[0015] Optionally, after replacing the oldest unit in the detection window with the latest request unit to form a new detection window, the method includes: calculating the sum of similarities of all request units in the new detection window after updating on each feature; updating the initial anomaly score of each request unit on each feature based on the updated sum of similarities; normalizing the initial anomaly score of each request unit to obtain the final anomaly score of each request unit based on the normalized initial anomaly score of each request unit, and determining normal request units and abnormal request units based on the final anomaly score of each request unit.

[0016] Optionally, the above-mentioned web server intrusion detection method further includes: receiving a partial reset command or a global reset command sent by a preset terminal; clearing the detection window of the corresponding API according to the partial reset command, or clearing the detection windows of all APIs according to the global reset command, to obtain a new version of the web application, so as to use the new version of the web application to collect new web requests and refill the detection window.

[0017] A second aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the web server intrusion detection method as described in the above embodiments.

[0018] A third aspect of this application provides a computer program product having a computer program stored thereon, which is executed by a processor to implement the web server intrusion detection method as described in the above embodiments.

[0019] In the above implementation, unit partitioning technology is used to divide web requests in the audit log into multiple request units. An API classification engine is used to identify the API affiliation of each request unit in real time based on a prefix tree algorithm. Based on the identification results, multiple request units are categorized by API to obtain multiple APIs. Each API maintains a detection window containing multiple request units. The feature similarity between multiple request units within each detection window is calculated, and the presence of abnormal request units deviating from the group behavior is identified based on the feature similarity. If abnormal request units deviating from the group behavior are found, a web server intrusion is detected, and the abnormal request units are removed. This solves the problems of existing web application anomaly detection technologies, such as limited applicability to web application anomaly detection scenarios, coarse detection granularity, lack of adaptive comparison benchmarks, and susceptibility to collaborative poisoning attacks. It eliminates the need for large-scale, time-consuming pre-training and frequent retraining, effectively adapts to complex monolithic web applications, maintains low false positives and high recall in heterogeneous behavior contexts, and is robust to advanced attack methods such as collaborative poisoning.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1This is a flowchart of a web server intrusion detection method provided according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the workflow of APIEcho according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram illustrating dynamic API classification according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram comparing the overall detection performance of APIEcho and a baseline method according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of similarity visualization results in a case study according to an embodiment of this application;

[0027] Figure 6 This is a schematic diagram comparing the performance of APIEcho and a baseline method under a poisoning attack according to an embodiment of this application;

[0028] Figure 7 This is a performance comparison diagram of APIEcho and the baseline method according to an embodiment of this application in an application update scenario;

[0029] Figure 8 This is a schematic diagram of the ablation experiment results of each component of APIEcho according to an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0032] The following describes a web server intrusion detection method according to an embodiment of this application with reference to the accompanying drawings. Addressing the problems mentioned in the background art regarding the limited applicability of existing web application anomaly detection technologies, such as coarse detection granularity, lack of adaptive comparison benchmarks, and vulnerability to coordinated poisoning attacks, this application provides a web server intrusion detection method. In this method, a unit partitioning technique is used to divide web requests in audit logs into multiple request units; an API classification engine is used to identify the API affiliation of each request unit in real time based on a prefix tree algorithm, and multiple request units are categorized by API according to the identification results to obtain multiple APIs. Each API maintains a detection window containing multiple request units; the feature similarity between multiple request units within each detection window is calculated, and the presence of abnormal request units deviating from the group behavior is identified based on the feature similarity; if abnormal request units deviating from the group behavior are found, a web server intrusion is detected, and the abnormal request units are removed. Therefore, it solves the problems of existing web application anomaly detection technologies, such as limited applicability to web application anomaly detection scenarios, coarse detection granularity, lack of adaptive comparison benchmarks, and susceptibility to collaborative poisoning attacks. It does not require large-scale, time-consuming pre-training and frequent retraining, can effectively adapt to complex monolithic web applications, maintain low false positives and high recall in heterogeneous behavior contexts, and is robust to advanced attack methods such as collaborative poisoning.

[0033] This application proposes APIECHO, a novel web server intrusion detection method designed specifically for monolithic applications, which can run without large-scale pre-training. The core insight of APIECHO is that legitimate requests to the same API endpoint exhibit highly similar underlying behavioral patterns. The system shifts the comparison granularity from replicas to individual requests within the same API, employing dynamic API classification, fine-grained behavioral feature extraction (including sequence and set-based features), adaptive similarity thresholds for each API, and a poisoning-resistant sliding window update mechanism.

[0034] Extensive evaluations across 16 real-world scenarios demonstrate that APIECHO significantly outperforms current state-of-the-art methods. It effectively adapts to application updates without retraining, is resistant to coordinated poisoning attacks, surpasses existing methods in average detection score, achieves over 90% attack recall, and maintains over 99% accuracy in detecting normal events. While delivering excellent detection performance, it also maintains low performance overhead, processing over 12,000 log events per second with less than 7GB of memory.

[0035] Anomaly detection in web applications aims to identify suspicious activities that deviate from normal behavior patterns, which is crucial for ensuring system security and defending against unknown attacks. Traditional anomaly detection methods can be mainly divided into two categories. The first category is rule-based verification, such as checking whether specific system call sequences or network connection patterns conform to preset security policies. This type of method is simple and intuitive, but its core drawback is that it cannot detect malicious behavior that exploits unknown vulnerabilities or employs new attack techniques, because these behavioral patterns are not included in existing rule bases.

[0036] The second type of approach attempts to build a detection model by learning normal behavior patterns. These methods typically collect data in a clean, normal operating environment and train a model that can represent the normal state of the system. When new behavior deviates significantly from this model, it is judged as abnormal. While these methods can detect some unknown attacks, they face a common challenge: the "normal behavior" of modern web applications, especially those that are frequently iterated and updated, is dynamically changing. Whenever an application undergoes feature upgrades, vulnerability fixes, or architectural adjustments, its underlying behavioral patterns (such as system calls and resource access) often change. This requires the detection model to be retrained to adapt to the new baseline of normal behavior; otherwise, a large number of false positives will occur. Retraining is not only time-consuming and labor-intensive, but in some scenarios, obtaining clean, "post-updated" normal data is itself a challenge.

[0037] To overcome the dependence on training baselines, researchers have proposed the ReplicaWatcher algorithm for microservice architectures. Its core idea is to leverage the highly similar behavior of multiple functionally identical replicas within a microservice architecture within the same time window. ReplicaWatcher detects anomalies by comparing the similarity of features such as the set of system calls made by different replicas within the same time window. If the behavior of a replica differs significantly from other replicas, it is considered that the replica may have been attacked. This self-comparison approach eliminates the need for retraining after application updates, as the behavior of all replicas changes synchronously, maintaining relative consistency. However, ReplicaWatcher also has its inherent limitations:

[0038] (1) Limited Applicability: The core assumption of ReplicaWatcher is the functional singularity and homogeneity of behavior among replicas. This makes it difficult to apply directly to traditional monolithic web applications. Replicas of monolithic applications typically carry multiple complex functions (e.g., handling user login, file upload, and backend management simultaneously), and the functional paths and underlying behaviors triggered by different requests may be drastically different. Even for normal requests, if they are scheduled to execute different functions on different replicas, ReplicaWatcher may generate false alarms due to the differences in behavior.

[0039] (2) Coarse-grained detection: ReplicaWatcher's detection unit is the aggregated behavior of the entire replica within a certain time window. This design not only limits its ability to pinpoint anomalies in a replica within a specific time period, but also restricts its feature extraction methods. Because it aggregates the behavior of all requests within the time window, ReplicaWatcher primarily relies on statistical features such as system call sets for comparison, making it difficult to capture subtle anomalies related to the order of behavior. For example, in a malicious request, the execution order of system calls may change, but the set of call types may not change significantly. This poses challenges for subsequent attack tracing and response, as well as for identifying specific types of attacks.

[0040] (3) Lack of adaptive comparison benchmark: ReplicaWatcher uses a uniform threshold for all types of behavior comparisons. However, the degree of difference in normal behavior between different functions can vary greatly. For example, the difference in system calls between two login requests may be much smaller than the difference between two requests to upload different files. A fixed threshold is difficult to adapt to this diversity of behavior, which may lead to false positives due to oversensitivity to some behaviors, or false negatives due to insufficient sensitivity to others.

[0041] (4) Vulnerable to coordinated poisoning attacks: If an attacker can launch similar malicious actions against all or most replicas of a service in the same time window, the behavioral differences between these replicas will be reduced, making it difficult for ReplicaWatcher to detect such coordinated attacks.

[0042] To address the aforementioned challenges, a novel web server intrusion detection method called APIEcho is proposed. The goal is to inherit the advantage of comparison methods that do not require large-scale pre-training, while overcoming the limitations of ReplicaWatcher when applied to monolithic web applications, and improving the accuracy and robustness of detection.

[0043] APIEcho's design is based on a key observation of web application behavior patterns: even in complex monolithic applications, multiple legitimate requests to the same API should exhibit a high degree of similarity in their underlying behavioral patterns. Based on this, the granularity of self-comparison is shifted from the replica level to different requests for the same type of API. Specifically:

[0044] (1) API-centric comparison, suitable for monolithic applications: Each request entering the system is precisely categorized into its corresponding API interface. This relies on a specially designed API classification engine that can dynamically identify URL parameters and adapt to the continuous evolution of APIs. In building this engine, the core ideas of the open-source project Akto were inspired, and a new implementation of its key API pattern matching and classification logic was developed based on a prefix tree, thus supporting dynamic updates of API templates and more flexible segmentation. This API-centric comparison approach naturally adapts to the diverse functionalities of different APIs in monolithic applications.

[0045] (2) Fine-grained comparison incorporating sequence features: The behavior of a single request is used as the basic unit for analysis and comparison. This fine-grained extraction of request behavior is achieved by employing the proposed unit partitioning technique, which can accurately segment and associate the mixed system-level audit logs with each Web request. This fine-grained processing not only locates anomalies to specific malicious requests but also enables the extraction of richer sequence behavior features of individual requests. This allows for the capture of the sequential relationships of system calls and local context information, thereby identifying covert attacks that merely change the order of behaviors without significantly altering the set of behaviors, thus compensating for the shortcomings of ReplicaWatcher, which relies solely on set features.

[0046] (3) Adaptive Comparison Benchmark: Since the behavior patterns of different APIs are inherently different (for example, the system call complexity and variability of the login API and the file upload API are completely different), the behavior similarity is calculated independently for each API's sliding window, and these similarities are adaptively normalized and thresholded using a batch normalization-like approach. In this way, each API has a "normal" benchmark that conforms to its own behavior characteristics, avoiding the problem of "one-size-fits-all".

[0047] (4) Addressing the issue of coordinated poisoning attacks: Detection and window updates are performed on a per-request basis. When a malicious request first appears, it is identified as an anomaly because its behavior is significantly different from existing normal requests of the same type within the window. Crucially, this anomaly-identified request is refused inclusion in subsequent sliding windows, thus preventing it from polluting the normal behavior baseline. This makes APIEcho more resistant to poisoning attacks, as ReplicaWatcher, after identifying an abnormal window, struggles to extract specific malicious behavior and prevent its impact on subsequent comparisons.

[0048] The overall workflow of APIEcho can be summarized as follows: First, it extracts behavioral units belonging to different user requests from audit logs through execution partitioning, and uses an API classification engine to categorize these units into specific APIs. Then, for each API, within a sliding window of its request units, it extracts the set and sequence features of each unit, calculates their similarity, and uses an adaptive detection algorithm to identify abnormal requests that deviate from the group behavior. Finally, based on the detection results and external notifications regarding application updates, it dynamically manages the sliding window state of each API to maintain detection sensitivity while adapting to normal application evolution and defending against potential poisoning attacks.

[0049] A series of experiments were conducted to comprehensively evaluate the effectiveness of APIEcho. Sixteen experimental scenarios were built using real web applications and vulnerabilities, and compared with state-of-the-art untrained methods like ReplicaWatcher and trained methods like Kairos and Provider. Experimental results show that APIEcho's overall detection performance significantly outperforms the three comparison methods, achieving an attack recall rate exceeding 90% in all scenarios and a benign event detection accuracy exceeding 99%. It is more effective than ReplicaWatcher in resisting coordinated poisoning attacks and adapts to application updates better than the three comparison methods without requiring retraining, maintaining detection performance. While achieving these excellent performances, APIEcho introduces only low computational and memory consumption, processing over 12,000 log events per second while maintaining a memory footprint of less than 7GB. Ablation experiments were also conducted to demonstrate the contribution of each design element in APIEcho to detection performance.

[0050] The main contributions of this application include:

[0051] (1) A novel Web server intrusion detection method that does not require large-scale pre-training is proposed. Its core idea is to compare different request behaviors of the same type of API, thereby overcoming the limitations of existing self-comparison methods in single applications.

[0052] (2) The APIEcho system was designed and implemented. The system integrates dynamic API classification, multi-dimensional feature extraction based on sequence and set, per-API adaptive threshold, and anti-poisoning sliding window update mechanism.

[0053] (3) A series of experiments were used to evaluate APIEcho. The results show that APIEcho can adapt to application updates without retraining and can effectively resist collaborative poisoning attacks. The results also show that in 16 scenarios using real web applications, APIEcho's detection performance surpasses state-of-the-art training-based methods, with attack recall rates exceeding 90% and benign event detection accuracy exceeding 99%.

[0054] Alice develops and maintains a complex monolithic web application based on the FastAPI framework, which is subject to continuous feature iteration.

[0055] First, Alice considered using an anomaly detection solution that required pre-training. An advanced anomaly detection system, Provdetector, was deployed, which identifies anomalies by learning a baseline model of system behavior. After initial deployment, Provdetector performed excellently, achieving a near 100% recall rate for known attacks while maintaining a high accuracy of 92.2%. However, after Alice performed a routine feature upgrade to its web application, the situation changed significantly. Due to the change in the application's underlying behavior patterns, Provdetector's original baseline model became inapplicable, leading to a surge in false positives and a sharp drop in accuracy to 54.1%. To restore detection performance, Provdetector had to be retrained. This process not only required the intervention of security experts but also relied on collecting large amounts of clean, normal traffic data and investing significant computing resources (such as GPUs) for hours, even exceeding a day, of model training. Considering the frequent updates required for Alice's application, such high operational costs and delayed response times were unacceptable.

[0056] Subsequently, Alice switched to a self-comparison method that does not require pre-training, attempting to use ReplicaWatcher. This method detects anomalies by comparing the behavioral similarities of different replicas of the same service within the same time window. However, because Alice's web application is a monolithic architecture hosting various heterogeneous functions, the types of requests handled and the functional paths executed by different replicas within the same time window can be drastically different. This inherent behavioral heterogeneity led to a large number of false positives for ReplicaWatcher. To reduce false positives, Alice had to significantly increase the detection threshold. While this improved the precision to 98.5%, the trade-off was a significant decrease in the system's sensitivity to real attacks, with a recall rate of only 50%.

[0057] More seriously, attackers could exploit vulnerabilities in ReplicaWatcher's mechanisms to launch coordinated poisoning attacks. When attackers simultaneously launch similar malicious actions against all or most copies of the application, the behavioral differences between the copies actually decrease, making it difficult for ReplicaWatcher to effectively identify such attacks. In a simulated coordinated poisoning scenario, even when Alice readjusted the detection threshold in an attempt to maintain a 55.0% recall rate, its precision further deteriorated to 51.4%, rendering it almost useless.

[0058] The aforementioned challenges clearly demonstrate that Alice urgently needs a new type of web server intrusion detection method, which should possess the following characteristics: 1) no need for large-scale, time-consuming pre-training and frequent retraining; 2) effective adaptation to complex monolithic web applications, maintaining low false positives and high recall in heterogeneous behavioral contexts; and 3) robustness against advanced attack methods such as coordinated poisoning. This is precisely the core problem that APIEcho aims to solve.

[0059] APIEcho primarily focuses on remote attacks launched against monolithic web applications via public network interfaces. Attackers typically exploit software vulnerabilities in web applications by crafting malicious HTTP requests to achieve objectives such as control flow hijacking, stealing sensitive data (e.g., through authentication bypass or directory traversal), compromising file system integrity (e.g., file inclusion exploits), or executing unauthorized code and operating system commands (e.g., command injection).

[0060] Similar to many other anomaly detection methods based on audit logs, APIEcho currently does not directly target attacks that do not involve or produce identifiable behavioral differences at the system call level, such as data tampering at the pure application logic level (if it does not trigger changes in underlying behavior). Furthermore, side-channel attacks, hardware-level vulnerability exploits (such as Spectre), resource exhaustion in denial-of-service attacks (such as network flooding, massive connection requests), remote passive system fingerprinting, various network spoofing attacks, and attack scenarios that require attackers to have prior physical access or have already penetrated the internal network are also not within the primary detection scope of the current method.

[0061] Assume the server operating system running the web application has reliable audit logging capabilities, capable of recording system-level behavioral information including process / thread identifiers, and that APIEcho can access and process these logs to categorize the behavior of each web request. Furthermore, assume the audit log generation mechanism and the APIEcho system itself are part of a trusted computing base, ensuring their integrity and availability during an attack, and preventing easy tampering or disabling by attackers. These are common and necessary assumptions for many security analysis and intrusion detection methods that rely on logs.

[0062] Figure 1 This is a flowchart illustrating a web server intrusion detection method provided in an embodiment of this application.

[0063] APIEcho's workflow can be organized into three parts, such as Figure 2 As shown:

[0064] Part 1: Request Segmentation and API Classification. In this part, APIEcho breaks down the application's audit log entries into units categorized by Web API.

[0065] 1) Unit Partitioning: First, APIEcho uses the unit partitioning method proposed in TeSec to introduce partitioning information into the audit logs by analyzing the execution model of the target web application and using different methods. Then, APIEcho uses the partitioning information to decompose the audit logs into different units, each unit corresponding to a specific web request.

[0066] 2) Dynamic API Classification: To identify which API each web request belongs to, APIEcho maintains an APICatalog based on a prefix tree algorithm, supporting real-time API classification. For a given segment, APIEcho extracts the URL of the web request corresponding to that segment and queries the APICatalog for its corresponding API; simultaneously, it updates the APICatalog using this URL to adapt to application updates and automatically correct outdated APIs.

[0067] Part Two: API Internal Anomaly Detection. For each API, APIEcho maintains a detection window containing several request units. By comparing the similarity between request units in the detection window, it outputs the detection result for the last unit in the window.

[0068] 1) Feature extraction: First, APIEcho extracts features from the internal audit log for each request unit within the window, including sequence features and set features, etc.

[0069] 2) Similarity-based anomaly detection: APIEcho calculates the similarity between extracted unit features and embeds them into anomaly vectors; then, anomaly detection algorithms are used to identify outliers and determine that the corresponding request units are anomalies.

[0070] Part 3: Window Management and Baseline Adaptation. APIEcho determines the content of the next detection window based on the detection results of the last unit. If a request is detected as malicious, it is removed from the detection window to ensure the accuracy of subsequent detections and defend against coordinated poisoning attacks; otherwise, the unit is retained within the detection window to accommodate drifting of normal behavior.

[0071] Specifically, this web server intrusion detection method includes the following steps:

[0072] In step S101, the web requests in the audit log are divided into multiple request units using unit partitioning technology.

[0073] Optionally, in some embodiments, the web requests in the audit log are divided into multiple request units using unit partitioning techniques, including: for multi-process concurrency models or multi-thread concurrency models, using the process ID and thread ID recorded in the audit log, grouping log entries by source, and inserting a first delimiter log before each worker process / thread starts processing a new request, so as to determine the request unit to which the log entry belongs based on the first delimiter log; for coroutine concurrency models, inserting a second delimiter log each time the web application switches coroutines, so as to determine the request unit to which the log entry belongs based on the second delimiter log.

[0074] The first phase of APIEcho is request segmentation and API classification. The goal of this phase is to transform the raw, massive stream of audit logs into a series of behavioral units associated with specific web requests and precisely categorized into corresponding API interfaces. This transformation process forms the basis for subsequent fine-grained anomaly detection, ensuring that comparisons and analyses are performed across comparable sets of behaviors. This phase primarily comprises two closely cooperating sub-modules: first, the unit segmentation module is responsible for accurately separating the behavioral fragments corresponding to individual web requests from the mixed audit logs; second, the dynamic API classification module further maps these behavioral units to specific API interfaces.

[0075] APIEcho's first step is to precisely segment the mixed system-level audit logs and associate each segment with a specific web request; this process is called cell segmentation. Accurate cell segmentation is the foundation for all subsequent analysis, ensuring that fine-grained modeling and comparison of behavior for individual requests are possible.

[0076] In implementing the unit partitioning, the audit log partitioning technology proposed by TeSec was adopted. This technology aims to solve two core challenges: first, how to associate low-level system behavior (such as system calls) with high-level network requests (i.e., user-initiated API calls); and second, how to handle the interleaving of log entries caused by the concurrency mechanism of the web server (multi-process / multi-thread, coroutines), ensuring that each partitioned unit contains only the behavior related to a single web request.

[0077] The core of TeSec's approach lies in cleverly inserting delimited logs into the audit logs. These special log entries are generated by a lightweight, instrumented web application at critical points in request processing and contain information sufficient to identify the specific backend request (e.g., a unique request ID).

[0078] The approach of this technology varies depending on the concurrency model:

[0079] 1. Multi-process / multi-threaded concurrency model: In this model, the web server typically maintains a process pool or thread pool. Each worker process / thread in the pool processes a series of requests sequentially. When a worker process / thread completes processing one request, it moves on to the next. Therefore, system calls triggered by the same process / thread are recorded sequentially in the audit log. Using the process ID (PID) and thread ID (TID) recorded in the audit log, interleaved log entries can be grouped according to their originating PID / TID.

[0080] To establish the association between system calls and specific requests, the method requires the web application to insert a "delimiter log" carrying a request identifier into the audit log before each worker process / thread begins processing a new network request. Thus, for any ordinary audit log entry, it is only necessary to trace back to the most recent "delimiter log" in the log sequence of its corresponding PID / TID to determine which network request the entry belongs to. Because log entries from different processes / threads are already distinguished by PID / TID, their interleaving does not affect the order and accuracy of the association within a single process / thread's logs.

[0081] 2. Coroutine-based Concurrency Model: Coroutines are a lightweight user-space concurrency mechanism. Unlike processes / threads, coroutine switching occurs in user space, and operating system audit logs typically do not record coroutine switching behavior. Therefore, PID / TID alone cannot distinguish the behavior of multiple coroutines executing concurrently within the same thread.

[0082] To address this, TeSec provides more granular instrumentation for web applications, inserting "delimiter logs" not only when request processing begins but also every time a coroutine switches. These "delimiter logs" also contain identifiers of the requests currently being processed by the coroutine. In this way, even within the same PID's log stream, these more frequently inserted "delimiter logs" can accurately separate mixed sequences of system calls into individual concurrent requests.

[0083] By employing TeSec's unit partitioning technique, APIEcho can effectively extract a precise behavioral unit (i.e., a set of ordered system call records) from the raw audit logs for each request entering the web server. This behavioral unit forms the basis for subsequent API classification, feature extraction, and anomaly detection. The advantage of this approach is that it requires very minimal modification to the web application; it can typically be integrated as a plugin without altering the application's core code, and it can complete log partitioning in linear time complexity.

[0084] In step S102, the API classification engine is used to identify the API affiliation of each request unit in real time based on the prefix tree algorithm, and multiple request units are classified into multiple APIs according to the identification results. Each API maintains a detection window containing multiple request units.

[0085] Optionally, in some embodiments, an API classification engine is used to identify the API affiliation of each request unit in real time based on a prefix tree algorithm, and multiple request units are classified into multiple APIs according to the identification results. This includes: constructing at least one API template using a prefix tree structure; using the HTTP request method and URL path of the request unit as the search key, when searching for a matching API template in the prefix tree, determining whether the search key matches the complete path in the prefix tree; if the search key completely matches the complete path in the prefix tree, and the leaf node of the complete path represents a defined API template, then the current request unit is classified as a defined API template; if there is a search key that does not completely match the corresponding API template, then the search key is recorded as a static key for parameterized template generalization.

[0086] After obtaining the behavioral units corresponding to individual web requests through unit segmentation, the next key step is dynamic API classification. The goal of this step is to accurately identify which of the many APIs provided by the application each request unit belongs to. To achieve this, especially in the context of frequent API iterations and evolutions in modern web applications, APIEcho employs a dynamic, prefix-tree-based API directory management and classification mechanism. Inspired by Akto's automatic API discovery, this mechanism can automatically learn and identify API structure from request URLs, including parameterized path segments, and adapt to API changes.

[0087] When a new request unit arrives, the system first attempts to match it with a known API template in the existing API directory. If the match fails, the request's URL path is treated as a new potential API endpoint and compared with other similar unknown endpoints in an attempt to discover a common API pattern (i.e., an API template containing parameters). Once a new API template is identified or an existing template is modified, the API directory is updated accordingly. Furthermore, this mechanism is adaptable to API changes and can handle API structure evolution due to application upgrades, such as previously fixed parameter path segments being replaced by multiple static paths.

[0088] Figure 3 This is an example of new request classification, parameter discovery, and prefix tree update.

[0089] API directory structure and request matching:

[0090] APIEcho uses a Trie data structure to implement its API directory. This tree structure is well-suited for efficiently storing and retrieving path-based API definitions. For example... Figure 3As shown in Figure (a), each node in the tree represents an HTTP request method (such as GET, POST) or a path segment of the URL (i.e., the part between two " / " in the URL). Starting from the root node, traversing the path representing the request method and each segment of the URL allows you to find the corresponding API.

[0091] Once a new request unit (already associated with a specific request in the previous stage) is generated, APIEcho extracts its HTTP method and the normalized URL path (typically with the hostname and query parameters removed) as a search key. This search key is then used for lookup in the prefix tree.

[0092] (1) Successful match: If the search key can completely match a path in the prefix tree, and the leaf node of that path represents a defined API template (which may contain parameterized nodes, such as those representing integers). <int>Nodes, such as Figure 3 In diagram (d), POST / api / other / <int>If the request unit is successfully categorized into this API, then the request unit is successfully classified. When matching parameterized nodes, the validation rules predefined by the parameter type are used (e.g., checking if the path segment is indeed an integer).

[0093] (2) Match failed: If the search key does not find an exact matching API template in the prefix tree (e.g., Figure 3 In diagram (b), when initially attempting to match POST / api / other / 2, the existing template is POST / api / other / 1 (which fails because the last two '2' do not match '1'). APIEcho temporarily records this complete, specific search key (such as POST / api / other / 2) as a static key. This indicates that a new, specific URL that has not yet been generalized into a template has been observed.

[0094] Optionally, in some embodiments, the search key is recorded as a static key and parameterized template generalization is performed, including: finding similar key pairs from the static key set that have the same HTTP method and the same number of URL segments as the search key and whose URL segments all conform to the same predefined parameter type; using the API template corresponding to the similar key pair as the API template of the search key, adding the API template of the search key to the prefix tree, and removing the search key covered by the generalized template.

[0095] Modern RESTful API designs widely employ URL path parameters (e.g., {userId} in / users / {userId} / profile). Treating each unique URL as a separate API is impractical, leading to an explosive growth in the number of APIs and failing to reflect the true structure of the API. Therefore, APIEcho's key capability lies in automatically discovering these parameters from collected static keys and generalizing specific URLs into API templates that include these parameters.

[0096] When a new static key is recorded (e.g.) Figure 3 In diagram (c), assuming POST / api / other / 1 and POST / api / other / 2 are recorded as static keys sequentially, APIEcho attempts to compare it with other similar static keys in storage to identify the variable parts as parameters. This process primarily relies on the following strategies:

[0097] (1) Predefined parameter type rules: A series of common parameter type recognition rules are predefined, such as numbers (Integer, Float), UUID (Universally Unique Identifier), hexadecimal strings, date / time stamps in specific formats, common file extensions (such as .jpg, .pdf), etc.

[0098] (2) Matching and generalization logic: When comparing two static keys (e.g., Figure 3 In diagram (c), comparing POST / api / other / 1 and POST / api / other / 2, if they have the same HTTP method and the same number of URL segments, and their different segments all conform to the same predefined parameter type (e.g., ... Figure 3 In diagram (c), if both 1 and 2 are integers, then APIEcho will consider these two static keys to belong to the same API template. Different segments will be generalized into a parameterized node (e.g., ...). Figure 3 In diagram (d), the generalization is... <int>).

[0099] (3) Handling Arbitrary String Parameters: For APIs with many path segments (e.g., more than one segment), allow at most one segment to be recognized as a parameter of type "arbitrary string". This helps handle cases where parameter values ​​do not have a fixed pattern (e.g., username, search keywords), as long as the structure of the rest of the API is consistent.

[0100] Once an API template is successfully generalized through the above comparisons (such as...), Figure 3 In diagram (d), POST / api / other / is generalized from POST / api / other / 1 and POST / api / other / 2. <int>This new API template will be added to the prefix tree. At the same time, the original static keys used for generalization (POST / api / other / 1, POST / api / other / 2) will be removed from temporary storage because they are now overridden by the more generic template. Nodes representing parameters in the prefix tree (such as...) <int>It will carry its type information for matching and verification of subsequent new requests.

[0101] Optionally, in some embodiments, the above-described web server intrusion detection method further includes: recording the parameter values ​​and timestamps of the parameterized URL segments of the API template; when the duration for which the number of parameter value types of any parameterized URL segment is less than a preset threshold is greater than a preset time, a new API path is generated for any parameterized URL segment and added to the prefix tree, and the API template of any parameterized URL segment is removed or modified.

[0102] Web applications evolve dynamically, and their API designs may change accordingly. For example, an API path that was originally distinguished by parameters (such as / items / {category_id}) may become several fixed paths (such as / items / electronics, / items / books) after an application update, each corresponding to different processing logic, making it unsuitable to represent with a single parameterized template. To address this situation, APIEcho designed an API splitting mechanism.

[0103] Parameter value monitoring: For each parameterized URL segment of the API template in the prefix tree, APIEcho records the actual observed parameter values ​​and the timestamp of each occurrence. Only parameter values ​​appearing within the most recent time window are considered to reflect the current state of the API. If the number of different values ​​observed for a parameterized segment remains below a preset minimum threshold over a period of time (e.g., only two or three fixed values ​​are observed for a long time), APIEcho considers that this segment may no longer be a parameter, but rather has been fixed to a few specific static values.

[0104] Execute splitting: When the splitting conditions are met, the original parameterized API template will be removed or modified. Instead, for each specific value recently observed on that URL segment, a new, more specific (potentially static or with a deeper parameter hierarchy) API path will be generated and added to the prefix tree. For example, if / items / {itemId} has only recently observed itemId as special-A and special-B, it might be split into two separate APIs (or path prefixes) / items / special-A and / items / special-B.

[0105] Through this dynamic API classification, template generalization, and splitting mechanism, APIEcho can effectively tag each request unit entering the system with an accurate API label, and maintain the accuracy and adaptability of classification even in monolithic applications where the API structure is constantly evolving.

[0106] In step S103, the feature similarity between multiple request units within each detection window is calculated, and the presence of abnormal request units that deviate from the group behavior in the detection window is identified based on the feature similarity.

[0107] In step S104, if there is an abnormal request unit that deviates from the group behavior, it is determined that a web server intrusion has been detected, and the abnormal request unit is removed.

[0108] Optionally, in some embodiments, calculating the feature similarity among multiple request units within each detection window, and identifying whether there are abnormal request units deviating from the group behavior in the detection window based on the feature similarity, includes: calculating the set feature similarity among request units using Jaccard similarity, and calculating the sequence feature similarity among request units using cosine similarity; calculating the initial abnormal score of each request unit on each feature dimension based on the set feature similarity and the sequence feature similarity; collecting the initial abnormal scores of all request units on each feature dimension to form a vector, and normalizing the vector to obtain a normalized abnormal feature vector; calculating the Euclidean distance between the normalized abnormal feature vector and the origin to obtain the final abnormal score of each request unit; if there is a target request unit with a final abnormal score greater than a preset value, then the target request unit is determined to be an abnormal request unit.

[0109] In some embodiments, the final anomaly score is:

[0110]

[0111] Among them, D i For requesting unit u i Final abnormal score, Let x' be the normalized anomalous feature vector. ki M represents the initial anomaly score of the normalized anomaly feature vector, where M and k are both the number of features.

[0112] After APIEcho successfully segments and categorizes audit log units to specific APIs, the next core task is to perform detailed anomaly detection on the request behavior within each API. APIEcho maintains a fixed-length detection window for each API interface in the system. When this window is filled with new request units, the system initiates a round of detection, aiming to identify abnormal individuals by comparing the behavioral similarities between various request units within the window, thereby discovering potential intrusion activities.

[0113] Since request units are essentially collections of audit logs, a complex data entity, they need to be converted into vectors. This mainly involves two steps: First, using domain knowledge and considering the operational characteristics of the web server, a set of rules was designed to extract features that reflect the core behavior of the unit. These features include set-based and sequence-based forms. Second, considering the goal of comparing units to identify behavioral deviations, the similarity between these extracted features was calculated, converting them into numerical representations, and further calculating anomaly scores for each unit. These scores, after normalization, are ultimately used to determine whether a unit is abnormal.

[0114] Further feature extraction is performed:

[0115] Similar to ReplicaWatcher, APIEcho extracts characteristics of units that cover different aspects of request processing behavior and are prone to change when compromised. First, system calls are the most fundamental way for programs to interact with the operating system kernel. When an intrusion occurs, to satisfy the attacker's objectives, the request unit is likely to execute system calls that are not normally executed. For example, if the attacker wants to execute a specific command, fork and exec system calls will be generated, which are uncommon in the request processing flow. Therefore, the set of system calls executed by the request unit and their categories is used as a feature. In particular, some more complex web server programs may execute various system calls during request processing, so that even if compromised, the above two characteristics do not change significantly. Therefore, the system call sequence generated during request processing is also directly used as a feature to improve detection capabilities in this case. Additionally, files and sockets are two main entities that interact with web server programs. Under normal circumstances, during request processing, the file and socket addresses accessed by the program should be within a relatively fixed range; however, when compromised, to steal data, collect information, or achieve remote control, the attacker may cause the program to access files outside the range or communicate with socket addresses outside the range. Therefore, features such as the accessed file directories, number of IP addresses, and number of port types were also extracted.

[0116] Further anomaly detection based on similarity:

[0117] APIEcho extracts three types of unit features: sets, sequences, and quantities. Set and sequence features still need to be converted into quantities or vectors before being input into anomaly detection algorithms. Since it only cares about whether a unit is similar to other units, and doesn't need to obtain the independent vector embedding of a particular unit, similar to ReplicaWatcher, it converts sets or sequences into quantities by calculating similarity to complete the embedding.

[0118] 1. Feature Similarity Calculation

[0119] For a specific API's detection window, let it contain N w A request unit, represented as U = {u1, u2, ..., u...} Nw For any two cells u in the window i and u j and the value F of their k-th feature. ki and F kj Calculate the similarity S between them. k (F ki ,F kj The similarity value range is [0,1].

[0120] 1) Set Features: For features such as sets of system calls or sets of file directories, Jaccard similarity is used to measure the similarity between them. For two sets A and B, the Jaccard similarity is defined as:

[0121]

[0122] Where |A∩B| represents the size of the intersection of sets A and B, and |A∪B| represents the size of their union.

[0123] 2) Sequence Features: For features such as system call sequences, the order information is crucial. Inspired by research on n-gram models in natural language processing, we first extract the frequency distribution of n-grams from the original sequence. An n-gram is a subsequence of n consecutive elements in a sequence. For example, for the system call sequence read, open, write, read, open, close, its 2-grams (bigrams) include (read, open), (open, write), etc. We treat the n-gram frequency distribution of a sequence (i.e., the relative frequency of each different n-gram in the sequence) as a high-dimensional vector, where each dimension corresponds to an n-gram that appears at least once in the two compared sequences, and the value in that dimension is the frequency of the corresponding n-gram. Then, we use cosine similarity to calculate the similarity between the two n-gram frequency distribution vectors, which serves as the similarity between the original sequences. For two vectors... and Cosine similarity is defined as:

[0124]

[0125] in, It is the dot product of vectors. It is a vector The L2 norm (Euclidean length) of .

[0126] Furthermore, construct the initial anomaly feature vector:

[0127] For each request unit u in the detection window U i (i∈{1,...,N w Construct an M-dimensional initial anomaly feature vector. Where M is the total number of features used. Each component x of the vector ki Representative unit u i The initial anomaly score on the k-th feature is calculated differently depending on the feature type:

[0128] (1) For set features and sequence features (let's call them the k-th features): their initial anomaly scores x ki It is obtained by calculating the complement of the average similarity between the feature of this unit and the corresponding features of all other units. Specifically:

[0129]

[0130] Among them, S k (F ki ,F kj ) is unit u i and u j Similarity on the k-th feature (using S) J or S C (Calculated). x ki The range of values ​​is [0,1]; the closer the value is to 0, the more it indicates the unit u. i The more similar the behavior of this feature is to the average behavior of other cells within the window, the more likely it is to be normal; conversely, the closer the value is to 1, the more likely it is to be abnormal.

[0131] (2) For numerical features (let's say the k-th feature, such as the number of IP addresses): its initial anomaly score x ki Calculate the difference between the feature value of this cell and the mean of this feature within the window. Let... Let be the mean of the k-th numerical feature within the current window, then:

[0132] x ki =F ki -μ k

[0133] This difference directly reflects the value of unit u. i The degree of deviation from this numerical characteristic.

[0134] Further normalization of feature dimensions:

[0135] After obtaining the initial abnormal feature vector of each unit Next, APIEcho normalizes the scores of all units within the detection window for each feature dimension. This step is inspired by the idea of ​​batch normalization. Its main purpose is to address the differences in behavioral variability between different APIs or features. For example, the difference in the set of directories accessed in two requests for a file upload API may naturally be greater than the difference in two requests for a simple status query API. If a uniform comparison scale is used for all features of all APIs, it is easy to become overly sensitive to some behavioral patterns (generating false positives) or undersensitive to others (generating false negatives). By normalizing the scores of each feature dimension within the current window, their distribution can be adjusted to a similar range, thus allowing the use of more comparable thresholds.

[0136] Specifically, for the k-th feature dimension, the initial anomaly scores of all units within the window are collected to form a vector. Then, L2Normalization is performed on this vector to obtain the normalized score.

[0137]

[0138] If the denominator is zero (i.e., all units have zero initial anomaly scores or differences on this feature), then x'ki is also zero. After this step, each requesting unit u i We obtain a normalized abnormal feature vector.

[0139] Finally, a final anomaly determination is performed:

[0140] Obtain the normalized abnormal feature vector Then, APIEcho calculates the Euclidean distance between the vector and the origin, which is used as the unit u. i The final abnormal comprehensive score D i :

[0141]

[0142] The overall score is D. i The larger the value, the greater the value of the unit u. i The greater the deviation of the behavior from the group behavior pattern of similar API requests within the current window, the higher the likelihood that it is an abnormal request.

[0143] Finally, the calculated D i Compare with a pre-set global threshold Θ. If D i >Θ, then determine the request unit u i This is abnormal.

[0144] Optionally, in some embodiments, the above-described web server intrusion detection method further includes: obtaining the latest request unit that recently entered the API in the detection window maintained by each API; replacing the oldest unit in the detection window with the latest request unit to form a new detection window; if the latest request unit is a normal request unit, then keeping the latest request unit in the new detection window and updating the API directory; if the latest request unit is an abnormal request unit, then discarding the latest request unit.

[0145] Furthermore, in some embodiments, after replacing the oldest unit in the detection window with the latest request unit to form a new detection window, the method includes: calculating the updated similarity sum of all request units in the new detection window on each feature; updating the initial anomaly score of each request unit on each feature based on the updated similarity sum; normalizing the initial anomaly score of each request unit to obtain the final anomaly score of each request unit based on the normalized initial anomaly score of each request unit, and determining normal request units and abnormal request units based on the final anomaly score of each request unit.

[0146] The final key component of APIEcho is its window management and baseline adaptation mechanism. This mechanism aims to ensure the detection system maintains accuracy and sensitivity as web applications evolve normally. Its core function is to dynamically maintain the detection window for each API, effectively resisting coordinated poisoning attacks and responding to external notifications about application updates, thereby achieving lightweight baseline reconstruction. This module ensures APIEcho's long-term effectiveness and low operational costs in real-world deployments.

[0147] APIEcho maintains a fixed-size N database for each API. w A sliding detection window. This window stores the N most recently entered into this API. w Each request unit. Once a new request unit is processed and feature extraction is complete, it enters the detection window of the corresponding API.

[0148] 1. Standard sliding and baseline drift adaptation: When a new requesting unit u new Once it arrives and is processed, it will replace the oldest request unit in the window. old This creates a new detection window. The content of this new window represents a "snapshot of normal behavior" for the API at this point. This continuous sliding update allows APIEcho to automatically adapt to gradual changes in the application's normal behavior (i.e., behavior drift), such as subtle changes in behavior patterns caused by minor code optimizations, configuration tweaks, or load variations.

[0149] 1) Cooperative poisoning attack defense and baseline protection: Based on the latest request unit within the current window. After anomaly detection is performed on the request (that was just added to the window):

[0150] 2) If If a request is deemed normal, it will remain in the window and serve as part of the normal baseline for subsequent comparisons. If the request brings new API pattern information (e.g., a new parameter format appears in the URL), the APICatalog can also be updated based on this normal request.

[0151] 3) If If deemed malicious, the system will issue an alert. Crucially, this request unit will be flagged as abnormal. The malicious request will be discarded from the window. Furthermore, any structural information carried by the malicious request will not be used to update the API catalog. In this way, the malicious request will not pollute the knowledge base representing "what is normal" for the API, effectively resisting coordinated poisoning attacks by attackers attempting to gradually "lower" the standard of normal behavior or "train" the system to accept malicious patterns by injecting a large number of similar malicious requests.

[0152] Further incremental similarity calculations are performed:

[0153] Simply put, after each window slide (i.e., an old cell leaves the queue and a new cell enters the queue), all N cells in the window are... w Each unit recalculates its complete anomalous feature vector (including all pairwise similarities and subsequent normalization), and its computational complexity may reach [value missing]. Where M is the number of features. To ensure the real-time performance of the system, especially in high-throughput environments, APIEcho employs an incremental computation method to update relevant metrics, optimizing the complexity of updating the anomaly feature vectors of all units within the window to O(N). w M).

[0154] Assuming the current detection window is u1 is the oldest unit. This is the latest unit. When the window slides, u1 is removed, and the new request unit u... new Joining creates a new window It is necessary to update each cell within the window (especially for cells). And the new unit u new Initial anomaly score x ki .

[0155] For unit The sum of their old similarities on feature k is:

[0156]

[0157] When the window is updated to W', for the same unit u i (It is still in W'), and its new sum of similarities is SumSim'. ki This can be achieved by subtracting the similarity to the removed unit u1 from the old sum and adding the similarity to the newly added unit u. new The similarity is used to obtain:

[0158]

[0159] This update applies to each cell remaining in the window (N in total). w -1) and each feature k requires constant time. Calculating all retained cells with u new The similarity needs to be O(N) w ).

[0160] For the newly added unit u new The sum of their similarities is SumSim' k,new Needs to be combined with other N in W' w -1 unit (i.e.) Calculate the similarity score for each item and sum them:

[0161]

[0162] This requires O(N) w (Number of similarity calculations)

[0163] Initial anomaly score x' in all units ki After the update / calculation is complete (based on the new SumSim' value), the L2 normalized denominator term for each feature dimension k will be... It also needs to be recalculated. Because all x' kj It is known that this step requires only O(N) time. w ).

[0164] Therefore, for M features, the overall update complexity of each window sliding is M×(O(N)). w )+O(N w ))=O(N w This allows APIEcho to efficiently handle continuous request streams.

[0165] Optionally, in some embodiments, the above-described web server intrusion detection method further includes: receiving a partial reset instruction or a global reset instruction sent by a preset terminal; clearing the detection window of the corresponding API according to the partial reset instruction, or clearing the detection windows of all APIs according to the global reset instruction, to obtain a new version of the web application, so as to collect new web requests using the new version of the web application and refill the detection window.

[0166] Modern web applications iterate rapidly, and legitimate code or architectural changes can lead to significant changes in their underlying behavior patterns. If the behavior of an API changes due to an update, APIEcho's original detection window and benchmarks may continue to misjudge the new, normal behavior as abnormal.

[0167] To address this situation, APIEcho provides a simple baseline reset mechanism:

[0168] When operations and maintenance personnel confirm that the continuous alerts of a certain API are due to application updates rather than actual attacks, they can notify APIEcho through an external notification mechanism.

[0169] Upon receiving a notification, APIEcho can clear the detection window for that specific API (or, in the case of a global update, clear the windows for all APIs).

[0170] Subsequently, the system will repopulate the window and build a new behavioral baseline based on the newly entered request.

[0171] This process is very lightweight, avoiding the need to collect a large amount of new logs and perform time-consuming and resource-intensive model retraining as in traditional methods, enabling APIEcho to quickly adapt to the normal state after application updates.

[0172] Most experiments were conducted on a Linux server with a 16-core Intel Xeon Platinum 8255C @ 2.50GHz CPU, 128GB of RAM, and running Ubuntu 22.04. Kairos experiments requiring GPU training were run on a separate server with an Intel Core i7-12700 @ 2.1GHz CPU, an RTX 3090 24GB GPU, 64GB of RAM, and running Kali Linux.

[0173] To evaluate APIEcho, as shown in Table 1, which lists the scenarios used for dataset generation and experimentation, experiments were conducted in 16 scenarios. To verify APIEcho's effectiveness on multi-version, multi-type, and multi-language web applications, the scenarios covered 16 versions of 8 web applications, including 10 real CVE vulnerabilities, and covering 4 common web application development languages ​​(Python, NodeJS, Java, and PHP). Automated scripts simulated various normal access behaviors to the applications, with each scenario's normal behavior lasting 20 minutes. After 10 minutes of normal traffic transmission, an attack was launched exploiting application vulnerabilities, resulting in audit log files containing a mixture of normal user behavior and attacker behavior, used for multiple experiments. Some of the 8 web applications required instrumentation unit partitioning; therefore, instrumentation unit partitioning was implemented for the following framework:

[0174] (1) FastAPI, an asynchronous web server framework for Python. This instrumentation implementation is applicable to python-demo and pgadmin, as well as other ASGI protocol web server frameworks that use the Python asyncio standard library;

[0175] (2) NodeJS, suitable for juice-shop and mongo-express and other web server applications that use NodeJS.

[0176] (3) Java, suitable for Java applications that use a thread pool execution model, such as Solr and ofbiz.

[0177] To evaluate the effectiveness of the experiment, malicious payloads were recorded during the attack. Since audit logs record detailed request URLs, matching these malicious payloads in the audit logs allowed the identification of system calls generated by the web application due to malicious requests, which were then marked as malicious.

[0178] Table 1

[0179]

[0180]

[0181] Further overall testing performance evaluation:

[0182] The overall intrusion detection performance of APIEcho was evaluated and compared with three other methods: ReplicaWatcher, ProvDetector, and Kairos. For ease of comparison, all audited events within the original alert unit of each method were considered malicious, thus unifying the detection granularity of the four methods to the event level. The recall rate of attack detection and the accuracy rate were recorded, and their harmonic mean (called the score) was used to compare the detection performance of each method. For each scenario, the algorithm was run 20 times to obtain the average detection performance.

[0183] How to implement the comparison method:

[0184] (1) For each scenario, first collect audit logs containing only benign behavior without attacking, and use them for training Provdetector and Kairos.

[0185] (2) Then, the mixed logs of malicious / benign behaviors with annotations are entered into four methods for intrusion detection.

[0186] (3) For ReplicaWatcher, multiple logs are collected at the same time. Each collected log is treated as a container replica and they are divided into time window inputs.

[0187] Experimental results are as follows Figure 4 As shown, Figure 4 This diagram illustrates the overall detection performance comparison between APIEcho and baseline methods. APIEcho detected over 90% of attacks across all scenarios, while its false positive rate for benign events was less than 1%. ReplicaWatcher, a training-free intrusion detection method that also employs a self-comparison approach, significantly outperformed APIEcho in both attack detection rate and benign event identification accuracy across all datasets.

[0188] ReplicaWatcher's core assumptions are functional singularity and behavioral homogeneity among replicas. However, in monolithic web application scenarios, this assumption doesn't hold if comparisons are simply made between replicas, which is the core reason for its poor performance. APIEcho, on the other hand, degrades the comparison granularity to the API level through API unit partitioning. Experimental results demonstrate that the assumptions of functional singularity and behavioral homogeneity hold true at this level. Kairos' overall detection performance is slightly worse than APIEcho, but it has fewer false positives in the Joomla scenario. However, Kairos requires log data containing only benign behavior to train its deep neural network model. Under the experimental scenarios and hardware conditions, the training time ranged from several hours to over a day. Provdetector's training time is shorter than Kairos', but its detection performance is also worse. In summary, the experimental data shows that APIEcho can achieve better detection performance than methods that require training in web server application scenarios without the need for training.

[0189] This application also uses case studies to analyze the differences in API behavior between the same web application and whether APIEcho can properly handle these differences in behavior.

[0190] Table 3 illustrates the behavioral characteristics of two APIs in the Python-Demo program. ` / solveapi` is responsible for parsing and executing expressions, while ` / statusapi` is responsible for checking the application's running status. Their normal behavior is shown in the first two columns of the table. ` / solveapi` has an arbitrary command execution vulnerability. When an attacker exploits this vulnerability to execute shell commands, the resulting behavior is shown in the third column of the table. As can be seen from the table, after being attacked, the behavior of ` / solveapi` differs significantly from its normal behavior, but is actually quite similar to the normal behavior of ` / statusapi`. If the features in the table are combined to calculate the similarity, resulting in an anomaly score vector, and then visualized using PCA, the results are as follows... Figure 5 As shown in Figure (a). Figure 5 Figure (a) shows that the data points of the abnormal score vector generated by the attack behavior of / solveapi are very close to some normal data points, which confirms the idea that the two behaviors are similar.

[0191] Table 3

[0192]

[0193]

[0194] As can be seen, for monolithic web applications, if the ReplicaWatcher approach is used without distinguishing between different APIs, the behavior of other APIs may interfere with the detection of abnormal behavior. APIEcho proposes an API classification scheme based on this point. If the similarity of the behaviors of the `solve` and `status` APIs is calculated separately, the resulting visualization is as follows: Figure 5 As shown in Figure (b), after removing the influence of other API behaviors through API classification, the abnormal behavior of / solveapi can be distinguished, proving the effectiveness of API classification.

[0195] The API classification scheme addresses the issue of significant differences in normal behavior between different APIs, leading to mutual interference in anomaly detection. However, depending on the functionality implemented by the API, the differences in normal behavior within different APIs may vary. This can result in the same distance threshold not being able to accurately detect different APIs simultaneously. Figure 5 As shown in Figure (c), the rainbowstone API of the ofbiz application exhibits a normal behavior that differs significantly from others (top right corner dot). Correctly identifying this normal behavior would lead to missed detections of the xmlrpcapi attack (bottom right red ×). To address this issue, APIEcho further normalizes the anomaly vectors for each API individually. This operation ensures that the distribution of anomaly vectors is nearly uniform across different APIs, regardless of the magnitude of the differences in their internal normal behaviors. Therefore, adaptive detection can be achieved across all APIs by specifying only a single detection threshold hyperparameter. Figure 5 As shown in Figure (d), after normalization, the normal behavior points of rainbowstoneapi are more concentrated, thus enabling the detection of attacks on xmlrpcapi without false positives.

[0196] In this section, we evaluate APIEcho's ability to resist coordinated poisoning attacks by simulating a poisoning attack.

[0197] First, we evaluated APIEcho's ability to combat coordinated poisoning attacks. Specifically, similar to RQ1, multiple log sets were still created. However, instead of a single attack, attacks were launched continuously at a frequency greater than or equal to normal access behavior after a period of time, simulating coordinated poisoning attack behavior. After obtaining audit logs containing the poisoning attack data, the detection results were obtained following the RQ1 process. The detection performance was compared with ReplicaWatcher, and the performance changes between the two methods compared to when they were not under poisoning attacks. Simultaneously, to verify the contribution of the mechanism of discarding abnormal units in the detection window against poisoning attacks, APIEcho-NoThrow, with this mechanism removed, was introduced for comparison.

[0198] Experimental results are as follows Figure 6 As shown, Figure 6 This diagram illustrates the performance comparison between APIEcho and the baseline method under poisoning attacks, with the ΔScore column recording the change in score compared to the results in RQ1. APIEcho scores higher than ReplicaWatcher and APIEcho-NoThrow in all eight scenarios. It's noteworthy that APIEcho's score is improved across all scenarios compared to the RQ1 results. This is because the proportion of attack behavior increases, allowing for parameter tuning to improve precision while maintaining recall. This also applies to the comparison methods. Even so, the detection performance scores of ReplicaWatcher and APIEcho-NoThrow still show a significant decrease compared to the results in RQ1 in some scenarios. The significant decrease in ReplicaWatcher's detection performance is due to its intrusion detection principle. Since ReplicaWatcher searches for outliers within the same batch of time windows through self-comparison, and in the simulated cooperative poisoning attack scenario, the proportion of attack behavior reaches over 50%, ReplicaWatcher has a high probability of misclassifying attack behavior as benign. APIEcho, which also uses a self-comparison method, faces the same problem. Therefore, a sliding window and a mechanism for discarding abnormal units were designed. By dynamically accumulating a portion of benign units in the sliding window, benign behavior is anchored. In the experimental results, APIEcho-NoThrow, which removes the abnormal unit discarding mechanism, showed a decrease in detection performance in some scenarios, demonstrating the effectiveness of this mechanism.

[0199] To further evaluate APIEcho's ability to resist poisoning attacks, a progressive poisoning attack was simulated to assess whether this approach could evade APIEcho's detection. High-frequency attack operations were performed, but initially, attacks that more closely resembled benign behavior (e.g., executing meaningless code for code injection vulnerabilities) were initiated. Then, genuine attack operations were gradually introduced, increasing their proportion. Since APIEcho defends against coordinated poisoning attacks by discarding anomalous units, if near-benign attacks could escape its initial detection and not be discarded, it could affect the detection window's recording of benign behavior, increasing the success rate of subsequent formal attacks.

[0200] A progressive poisoning attack experiment was conducted in an OFBiz scenario. Specifically, the attack method in the OFBiz scenario involves injecting arbitrary Groovy code into XML, with the original attack payload executing shell commands. In this experiment, as shown in Table 4, which lists the payloads used in the progressive poisoning attack experiment, the attack payload first executes meaningless print code, then reads some files that the OFBiz application might normally read, then accesses the directory that the attack payload will access, and finally executes the original attack payload. The APIEcho sliding window size was set to 100 in the experiment. The attack payload switching rounds were configured to ensure that if an attack behavior enters the detection window at a certain step, the unit of that behavior can effectively accumulate in the window before switching to the next payload. In this experiment, APIEcho still correctly identified 99.9% of benign events. This means that APIEcho's anomaly detection mechanism is sensitive enough that even a deliberately designed progressive attack will be detected at some step and defended against by discarding abnormal units.

[0201] Table 4

[0202]

[0203]

[0204] Further adaptation to application updates:

[0205] Evaluate APIEcho's ability to maintain detection performance after application updates. For each application scenario, a new application version is created by introducing new vulnerabilities or new normal behaviors. Audit logs of the new application version are collected and then concatenated with logs of the old version obtained in RQ1, simulating a scenario where the application is dynamically updated during real-time intrusion detection runtime. The detection performance of various methods on concatenating audit logs is compared. In particular, for Kairos and Provdetector, training is performed using only the normal behavior logs of the old version.

[0206] Experimental results are as follows Figure 7 As shown in Figures (a) and (b), Figure 7 This diagram compares the performance of APIEcho and the baseline method in application update scenarios. It shows that compared to the experimental results in RQ1, ReplicaWatcher, Kairos, and Provdetector exhibit significant performance degradation on multiple datasets. In this experiment, the Kairos and Provdetector models were trained using logs from older versions, which may prevent them from accurately fitting the normal behavior of the new version. This is the core reason for their performance degradation. ReplicaWatcher, on the other hand, progressively adapts to version updates through time window updates. Theoretically, if a time window happens to include both the normal behavior of the old and new versions, one of them may be judged as abnormal, affecting detection performance. However, on some datasets, the scores of the three comparison methods did not decrease significantly and even improved. Possible reasons for this phenomenon include:

[0207] 1. Different application versions exhibit varying behaviors. If the behavioral differences between two versions are small, the older version model may still be able to fit the behavior of the newer version.

[0208] 2. For some scenarios, different attacks were used in the new version, and the new attacks may be more obvious to their models.

[0209] Unlike the comparison methods, APIEcho did not show a significant decrease in detection performance on any dataset. APIEcho clears its detection window upon receiving an update notification. Experimental results show that this mechanism helps it refit the behavior of the new version, avoiding the influence of the old version and thus improving detection performance.

[0210] In the aforementioned experiments, there was a relatively long time interval between the application's version update and the next attack, sufficient for APIEcho to accumulate normal behavior of the new version within the detection window. To further evaluate the impact of APIEcho's mechanism of clearing the detection window upon receiving an update notification on detection performance, the following experiment was designed: the logs of the updated version in the previous experiments were adjusted so that attack behavior was listed first, simulating a scenario where the application was attacked immediately after a version update. Theoretically, APIEcho would then perform detection after the subsequent benign behavior units filled the detection window, thus ensuring detection accuracy. This experiment was conducted in the python-demo scenario. The results show that APIEcho can indeed detect all attacks, and the accuracy of benign event detection does not decrease. This proves that even if the application is attacked immediately after a version update, APIEcho's detection performance will not be affected; however, since a certain number of requests are needed to accumulate the detection window, alerts may be delayed.

[0211] Further ablation experiments and parameter sensitivity experiments were conducted:

[0212] Normalization mechanism. APIEcho uses normalization to adapt to the different internal behaviors of different APIs, allowing the same detection threshold to be applied to different APIs. To further evaluate the impact of this mechanism on detection performance, intrusion detection performance with normalization disabled was tested in some scenarios, and the results are as follows. Figure 8 The APIEcho-NoNormalize column in the middle. Figure 8 The results of ablation experiments on various components of APIEcho are shown. It is evident that disabling normalization has a significant impact on detection performance for applications with complex functions and more diverse API behaviors, such as ofbiz and solr, while the impact is smaller for applications with simpler API behaviors. This demonstrates that the normalization mechanism is beneficial to detection performance, especially in scenarios with diverse API behaviors.

[0213] Sequence Features. To detect more covert attacks, APIEcho introduces system call sequence features. Sequence feature extraction was disabled in certain scenarios, and the detection performance was tested under these conditions. The results are as follows: Figure 8 The APIEcho-NoSeq column shows that, contrary to disabling normalization, disabling sequence features significantly reduces detection performance in simple application scenarios like python-demo. This might be because if the request processing flow is simple, the system call sequence will be shorter and more fixed, making it less likely for n-gram combinations identical to the attack sequence to appear. Overall, system call sequence features are beneficial for detection performance.

[0214] API classification. APIEcho handles the differences in normal behavior between different APIs through API classification. This means that incorrect API classification can negatively impact intrusion detection performance. It was found that the probability of API extraction based on parameter type matching is very low; in most cases, API classification errors arise from arbitrary string parameter rules. Since there are no parameter type requirements, this rule may cause several APIs with only a single URL difference to be merged into one, with the different part of their URLs being treated as a parameter. To evaluate the impact of API classification errors on detection performance, a different number of API classification errors were artificially created in the OFBiz scenario. This scenario was chosen because it has a large number of APIs (approximately 40). The API category log extracted by APIEcho was exported, and the situation of API classification errors was simulated by artificially merging one or more APIs that conform to the above description. After importing the erroneous API category log into APIEcho, its accuracy was observed. Table 5 shows the results of this experiment, with the accuracy in the OFBiz scenario showing the presence of API classification errors. It can be seen that the accuracy only decreased once, when two APIs were merged for the first time. Therefore, in most cases, API classification errors do not significantly affect detection performance; and due to the reliability of parameter type matching and the rule that any string parameter can only have one such parameter, API classification errors that can seriously affect detection performance are difficult to occur.

[0215] Table 5

[0216] #Merged APIs 0 2 4 8 12 24 Accuracy% 99.5％ 99.3％ 99.3％ 99.3％ 99.3％ 99.3％

[0217] Sliding window size. APIEcho uses a sliding window as the unit for self-comparison detection. In each round of detection, the sliding window moves forward to include a new unit, and the similarity of all units within the window, including the new unit, is calculated to perform anomaly detection and determine whether the new unit is abnormal. Theoretically, increasing the sliding window size reduces the proportion of new attack behavior units in the window and reduces the impact on the similarity between other units, which should improve detection performance; however, the computational cost of detecting each new unit increases linearly with the window size. Therefore, experiments were conducted on the Solr scenario, adjusting the sliding window length while keeping other parameters constant, and observing the change in the proportion of correctly identified benign events. The experimental results are shown in Table 6, which shows the accuracy in the Solr scenario under different detection window lengths. The results show that the accuracy drops significantly when the window size is 10 compared to the default value (100); however, when the window size is increased to 50, the accuracy is basically the same as the default value, and the window size must be increased to 500 to achieve a slight improvement in accuracy. Therefore, considering the performance overhead of a larger window, the default window size of 100 is more appropriate.

[0218] Table 6

[0219] Window Size 10 50 100 (Default) 200 500 Accuracy% 95.0％ 98.3％ 98.3％ 98.3％ 98.4％

[0220] Performance overhead:

[0221] This section evaluates the performance overhead of APIEcho. Theoretically, the configurable parameter with the greatest impact on APIEcho's performance overhead is the size of the sliding window. The overhead of APIEcho in detecting each new unit increases linearly with the size of the sliding window. This is because when adding a new unit and removing an old unit, both updating the anomaly vectors of existing units based on the new unit and calculating the anomaly vectors of the new unit based on existing units are computationally complex, and the computational complexity of both is linearly related to the window size. Furthermore, the maximum memory occupied by the sliding window is also clearly linearly related to its size. Therefore, this section primarily evaluates the relationship between the sliding window size and APIEcho's memory consumption and detection throughput through experiments. Experiments are conducted in the ofbiz scenario, where the application has a large number of APIs. Since the sliding window for each API is independent, the changes in memory usage resulting from changing the sliding window length will be more significant.

[0222] Experimental Results. Detection was performed under different window sizes, and the average values ​​of memory consumption and log throughput (i.e., the number of log events processed per second) were recorded. As shown in Table 7, Table 7 shows the memory usage and log throughput of APIEcho under different sliding window lengths. It can be seen that although memory usage and log throughput do increase and decrease respectively with increasing window size, this trend gradually slows down. Based on the evaluation of the relationship between sliding window size and detection performance, choosing a window size of 100 is more appropriate.

[0223] Table 7

[0224]

[0225] 1) Cold start issue. After initial deployment or clearing the window, APIEcho needs time to learn and build the APICatalog and fill the detection windows of each API with normal request units before it can start effectively detecting anomalies. For web applications with low user traffic, or for certain API interfaces that are not frequently called in the application, this "cold start" phase may be relatively long, resulting in limited detection capabilities for these APIs before the windows are filled.

[0226] 2) Reliance on external notifications to adapt to API changes. When web application version upgrades or architectural adjustments cause significant and legitimate changes in API behavior patterns, APIEcho currently relies on external notifications to trigger the clearing of the corresponding API detection window and baseline reconstruction. This is because the system itself cannot fully autonomously distinguish between such legitimate behavioral upheavals and continuous, slow-moving coordinated poisoning attacks. Nevertheless, APIEcho's adaptation process upon receiving notifications is automated and far more lightweight and efficient than traditional methods that require collecting large amounts of clean data and performing large-scale model retraining. In practical deployments, integration with CI / CD systems can be considered to automatically acquire application update signals, further reducing manual intervention.

[0227] 3) Potential diversity in behavior within the same API. One of APIEcho's core assumptions is that multiple legitimate requests to the same API should exhibit highly similar behavior. However, in some complex web applications, some APIs may execute vastly different internal logic and underlying operations depending on the input parameters. For example, a general data processing API may have drastically different system call sequences and resource access patterns depending on the operation type specified by the parameters (such as read, write, delete, or transform). In such cases, even after APIEcho's per-API normalization, these normal requests originating from the same API but exhibiting drastically different behaviors may make the benchmark for "normal" behavior too broad, thereby reducing the sensitivity to detect subtle anomalies and potentially making it difficult to effectively identify attacks targeting that API.

[0228] Research on this topic can be broadly categorized into rule-based methods and machine learning-based methods. Rule-based methods, such as HOLMES, MORSE, and CONAN, rely on predefined detection rules based on domain knowledge. HOLMES enhances the interpretability of results by constructing a high-level scenario graph to correlate low-level logs with high-level attack tactics. MORSE employs a label propagation mechanism, spreading sensitive or suspicious labels in the source graph according to predefined rules to identify anomalies. CONAN, on the other hand, builds a state machine model based on the phased characteristics of APT attacks, enabling lightweight attack detection. These methods have the advantage of relatively low false positive rates and ease of understanding, but their core drawback is the difficulty in detecting malicious behavior exploiting unknown vulnerabilities or novel attack techniques, as these behavioral patterns are not included in existing rule bases.

[0229] To overcome this limitation, machine learning-based methods have been proposed, which build detection models by learning normal system behavior patterns, such as GNN-based and RNN-based models. These methods can generally detect unknown attacks, but they face a common challenge: when application updates cause changes in normal behavior patterns, the model needs to be retrained, which is not only time-consuming and labor-intensive, but may also make it difficult to obtain clean "post-update" normal data. ReplicaWatcher proposes a training-free self-comparison method for microservice architectures, which discovers anomalies by comparing the behavioral similarity of different replicas within the same time window, thus eliminating the need for retraining after application updates. However, ReplicaWatcher suffers from limitations in its applicability to complex monolithic web applications, including coarse detection granularity, lack of adaptive comparison benchmarks, and vulnerability to collaborative poisoning attacks.

[0230] To address the shortcomings of traditional methods, particularly the dependence on training baselines and the limitations of ReplicaWatcher in monolithic applications, APIEcho proposes a new approach. By comparing different request behaviors of similar APIs, it inherits the advantage of self-comparison methods that do not require large-scale pre-training, while overcoming the aforementioned challenges, improving detection accuracy and adaptability to monolithic applications.

[0231] To achieve accurate analysis of individual web request behavior, APIEcho needs to precisely segment and correlate the mixed system-level audit logs to each web request. This process faces the so-called "dependency explosion" problem, where the correspondence between log events generated by long-running web application server processes handling a large number of concurrent requests becomes difficult to clarify. Early work, such as BEEP, first proposed the idea of ​​dividing long-running applications into multiple execution units for accurate source tracing. Subsequent research, such as MPI, proposed methods based on code data structure annotations and memory dependency tracking, mainly addressing unit partitioning in thread pool scenarios, but requiring high levels of expertise from security specialists and lacking intuitiveness in modern asynchronous programming models. OmegaLog and WinLog can automate multi-process and multi-threaded scenarios, but lack sufficient support for asynchronous event loops. APIEcho borrows the unit partitioning technique proposed in TeSec. TeSec provides a detailed classification, discussion, and adaptation for the asynchronous execution models widely used in web applications (including multi-process, multi-thread, thread pool, and event loop). It can effectively automate the problem of accurately attributing underlying behaviors such as system calls to specific web requests under these complex concurrency modes, providing key technical support for APIEcho to achieve fine-grained request behavior extraction and comparison.

[0232] In terms of API identification and discovery, existing work employs various strategies. Some tools infer APIs from browser behavior, such as the Akita Software Chrome Extension, which records API activity when users browse websites as a Chrome extension. It can automatically detect endpoints, generate searchable API views, and export the OpenAPI v3.0 specification, facilitating developers' understanding and documentation of APIs. Other tools rely on existing API specifications for deeper probing and crawling, such as ZAP and Kiterunner. ZAP requires obtaining the OpenAPI specification before crawling and then explores APIs within a defined scope. Kiterunner, on the other hand, builds its own pattern library by organizing and compressing a large Swagger specification dataset and systematically generates requests to test API endpoints based on it. In addition, there is a category of passive API discovery tools that identify APIs by monitoring actual network traffic. For example, Cloudflare's API Gateway (requires its service) uses machine learning to dynamically discover APIs and generate OpenAPI specifications. Its methods include using session identifiers such as HTTP headers and cookies, combined with common path patterns (such as containing " / api / " or version numbers) for API traffic identification. The open-source tool Akto also builds an API inventory by capturing traffic between clients and servers and directly uses the inferred API requests for security testing without pre-generating API specifications. APIEcho's API classification engine was inspired by Akto's core idea of ​​passively discovering APIs. It reimplements its key API pattern matching and classification logic based on a prefix tree and introduces a corresponding update mechanism to adapt to the continuous evolution of APIs when they change.

[0233] This application proposes APIEcho, a novel web server intrusion detection method that does not require large-scale pre-training. It aims to overcome the limitations of existing self-comparison methods on monolithic web applications and improve detection accuracy and robustness. The core mechanism of APIEcho is to move the comparison unit down from the replica level to different requests of the same API. Through a specially designed dynamic API classification engine, fine-grained request behavior extraction based on unit partitioning, per-API adaptive normalization thresholds, and a poisoning-resistant sliding window update mechanism, it effectively solves the dependence of traditional methods on training baselines and the shortcomings of ReplicaWatcher in terms of applicable scenarios, detection granularity, comparison benchmarks, and collaborative poisoning defense. Using an experimental scenario containing 16 real web applications and vulnerabilities, APIEcho is compared with the state-of-the-art non-trained method ReplicaWatcher and trained methods Kairos and ProvDetector. Experimental results show that APIEcho can better adapt to application updates without retraining, effectively resist collaborative poisoning attacks, and significantly outperforms comparative methods in all scenarios, with an attack recall rate exceeding 90% and a benign event detection accuracy exceeding 99%. Simultaneously, it maintains low computational and memory consumption, processing over 12,000 logs per second with a memory footprint of less than 7GB. The implementation is open source.

[0234] It should be noted that the API parameter types and validation rules in this application are shown in Table 8. Table 8 lists the parameter types used in the API catalog and their validation rules used in template generation and trie matching.

[0235] Table 8

[0236]

[0237] The features extracted from the request unit for anomaly detection are shown in Table 9.

[0238] Table 9

[0239]

[0240]

[0241] The web server intrusion detection method proposed in this application utilizes unit partitioning technology to divide web requests in audit logs into multiple request units. Using an API classification engine, based on a prefix tree algorithm, the API affiliation of each request unit is identified in real time. Based on the identification results, multiple request units are categorized by API to obtain multiple APIs. Each API maintains a detection window containing multiple request units. The feature similarity between multiple request units within each detection window is calculated, and based on the feature similarity, it is identified whether there are abnormal request units deviating from the group behavior within the detection window. If abnormal request units deviating from the group behavior are found, a web server intrusion is detected, and the abnormal request units are removed. This solves the problems of existing web application anomaly detection technologies, such as limited applicability to web application anomaly detection scenarios, coarse detection granularity, lack of adaptive comparison benchmarks, and susceptibility to collaborative poisoning attacks. It eliminates the need for large-scale, time-consuming pre-training and frequent retraining, effectively adapts to complex monolithic web applications, maintains low false positives and high recall in heterogeneous behavior contexts, and is robust to advanced attack methods such as collaborative poisoning.

[0242] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0243] The memory 901, the processor 902, and the computer program stored on the memory 901 and capable of running on the processor 902.

[0244] When processor 902 executes the program, it implements the Web server intrusion detection method provided in the above embodiments.

[0245] Furthermore, electronic devices also include:

[0246] Communication interface 903 is used for communication between memory 901 and processor 902.

[0247] The memory 901 is used to store computer programs that can run on the processor 902.

[0248] The memory 901 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0249] If the memory 901, processor 902, and communication interface 903 are implemented independently, then the communication interface 903, memory 901, and processor 902 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0250] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0251] The processor 902 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0252] This application also provides a computer program product on which a computer program is stored, which, when executed by a processor, implements the above-described Web server intrusion detection method.

[0253] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0254] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0255] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0256] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be specifically implemented in any computer program product for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer program product" can be any means that can contain, store, communicate, propagate, or transmit a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer program products include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, the computer program product can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0257] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0258] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer program product, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0259] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer program product.

[0260] The computer program product mentioned above may be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.< / int> < / int> < / int> < / int> < / int>

Claims

1. A method of intrusion detection for a Web server, characterized by, The method comprises the following steps: dividing web requests in an audit log into multiple request units by using a unit division technique; using an API classification engine to identify the API attribution of each request unit in real time based on a prefix tree algorithm, and classifying the multiple request units into multiple APIs according to the identification result, wherein each API maintains a detection window containing multiple request units; calculating the feature similarity between multiple request units in each detection window, and identifying whether there is an abnormal request unit with deviated group behavior in the detection window according to the feature similarity; if there is an abnormal request unit with deviated group behavior, it is determined that a web server intrusion is detected, and the abnormal request unit is removed.

2. The method of claim 1, wherein, The calculation of the feature similarity between multiple request units in each detection window and the identification of whether there is an abnormal request unit with deviated group behavior in the detection window according to the feature similarity comprise: calculating the set feature similarity between request units by using Jaccard similarity, and calculating the sequence feature similarity between request units by using cosine similarity; calculating the initial abnormal score of each request unit in each feature dimension based on the set feature similarity and the sequence feature similarity; collecting the initial abnormal scores of all request units in each feature dimension to form a vector, and normalizing the vector to obtain a normalized abnormal feature vector; calculating the Euclidean distance between the normalized abnormal feature vector and the origin to obtain the final abnormal score of each request unit; if there is a target request unit with a final abnormal score greater than a preset value, the target request unit is determined to be an abnormal request unit.

3. The method of claim 1, wherein, The division of web requests in an audit log into multiple request units by using a unit division technique comprises: for a multi-process concurrent model or a multi-thread concurrent model, grouping log entries by source by using process IDs and thread IDs recorded in the audit log, and inserting a first delimiter log before each working process / thread starts processing a new request to determine the request unit to which the log entry belongs according to the first delimiter log; for a coroutine concurrent model, inserting a second delimiter log each time the web application switches a coroutine to determine the request unit to which the log entry belongs according to the second delimiter log.

4. The method of claim 1, wherein, The use of an API classification engine to identify the API attribution of each request unit in real time based on a prefix tree algorithm and to classify the multiple request units into multiple APIs according to the identification result comprises: constructing at least one API template by using a prefix tree structure; when searching for a matching API template in the prefix tree by using the HTTP request method and URL path of a request unit as a search key, judging whether the search key matches a complete path in the prefix tree; if the search key completely matches a complete path in the prefix tree, and the leaf node of the complete path represents a defined API template, the current request unit is classified into the defined API template, and if there is a search key that does not completely match a corresponding API template, the search key is recorded as a static key for parameterized template generalization.

5. The method of claim 4, wherein, The recording of the search key as a static key, parameterized template generalization, comprises: finding a similar key pair from the static key set, which has the same HTTP method and URL segment number as the search key and the URL segments of which conform to the same predefined parameter type; taking the API template corresponding to the similar key pair as the API template of the search key, adding the API template of the search key to the prefix tree, and removing the search key covered by the generalized template.

6. The method of claim 5, wherein, Further comprising: recording the parameter values and timestamps of the parameterized URL segments of the API template; when the number of parameter values of any parameterized URL segment is less than the preset threshold for a duration greater than the preset time, generating a new API path for the any parameterized URL segment and adding it to the prefix tree, and removing or modifying the API template of the any parameterized URL segment.

7. The method of claim 2, wherein, The final anomaly score is: where D i is the request unit u i the final anomaly score, is the normalized anomaly feature vector, x' ki is the initial anomaly score of the normalized anomaly feature vector, M and k are both the number of features.

8. The method of claim 1, wherein, Further comprising: obtaining the latest request unit of each API maintained detection window that enters the API; replacing the oldest unit in the detection window with the latest request unit to form a new detection window; if the latest request unit is a normal request unit, retaining the latest request unit in the new detection window and updating the API directory, and if the latest request unit is an abnormal request unit, discarding the latest request unit.

9. The method of claim 8, wherein, After replacing the oldest unit in the detection window with the latest request unit to form a new detection window, comprising: calculating the updated similarity sum of all request units in the new detection window on each feature; updating the initial anomaly score of each request unit on each feature based on the updated similarity sum; normalizing the initial anomaly score of each request unit to obtain the final anomaly score of each request unit according to the normalized initial anomaly score of each request unit, and determining the normal request unit and the abnormal request unit according to the final anomaly score of each request unit.

10. The method of claim 1, wherein, Further comprising: receiving a local reset instruction or a global reset instruction sent by a preset terminal; emptying the detection window of the corresponding API according to the local reset instruction, or emptying the detection windows of all APIs according to the global reset instruction to obtain a new version of the web application, and collecting new web requests using the new version of the web application to refill the detection window.

Images