Abnormality detection method, device, equipment and medium
By dividing a metabolized window in the API call record, durable co-modulation features are extracted, and using the SVM model for detection, the accuracy and adaptability of API anomaly detection in the prior art is solved, and efficient and accurate API anomaly detection is achieved.
Patent Information
- Application Number
- CN202411595533.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-04-18
AI Technical Summary
It is difficult to accurately and effectively detect whether API calls are abnormal, especially when facing complex and changeable API calls patterns and advanced attack techniques.
By obtaining the API call records within the preset time period, dividing them into sub-windows, calculating the number of calls, call success rate, and average response time of the API, and determining its persistent co-tuning characteristics. This data is input to the pre-trained Support Vector Machine (SVM) to identify the model to obtain the detection results of whether the API is abnormal.
It realizes accurate detection of whether the API is abnormal without relying on predefined rules, improves detection accuracy and adaptability, and can effectively identify complex API calling patterns and advanced attack behaviors.
Smart Images
Figure CN119848639A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data security technology, and in particular to an anomaly detection method, device, equipment and medium. Background Art
[0002] With the rapid development of the Internet and mobile Internet, Application Programming Interface (API) has become an important bridge for communication and data exchange between applications and is widely used in various systems and services. API call logs record detailed information about each API request. With the increase in API call frequency and the increase in system complexity, the data size of API call logs is also growing rapidly. How to effectively analyze and process this data, especially how to detect API anomalies, has become an important research topic.
[0003] Traditional API anomaly detection methods mainly rely on rule matching and statistical analysis. These methods usually identify abnormal behaviors through predefined rules or thresholds, such as a sudden increase in the call frequency of a certain API or a value of a certain parameter outside the normal range. However, these methods often show limitations and inaccuracies when faced with complex and changeable API call patterns. In particular, when attackers use advanced attack techniques (such as low-frequency and slow attacks, parameter pollution attacks, etc.), traditional methods are difficult to effectively identify. Summary of the invention
[0004] The embodiments of the present application provide an anomaly detection method, apparatus, device and medium for solving the problem in the related art that it is impossible to accurately and effectively determine whether an API call is abnormal.
[0005] In a first aspect, an embodiment of the present application provides an anomaly detection method, the method comprising:
[0006] Obtain each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful;
[0007] Divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine a persistent homophonic feature of the point set;
[0008] Each of the call data and each persistent homology feature is input into a pre-trained support vector machine (SVM) recognition model, and a detection result of whether the API is abnormal output by the SVM recognition model is obtained.
[0009] In a second aspect, an embodiment of the present application further provides an abnormality detection device, the device comprising:
[0010] An acquisition module is used to acquire each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes a response time and a result of whether the call is successful;
[0011] A processing module is used to divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine the persistent homology features of the point set; input each call data and each persistent homology feature into a pre-trained SVM recognition model, and obtain a detection result of whether the API is abnormal output by the SVM recognition model.
[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including:
[0013] A memory for storing program instructions;
[0014] The processor is used to call the program instructions stored in the memory and execute the steps included in any one of the above-mentioned abnormality detection methods according to the obtained program instructions.
[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for anomaly detection as described in any one of the above items.
[0016] In an embodiment of the present application, an electronic device obtains each call record of an API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful; the preset time period is divided into sub-windows of preset time length, and for each sub-window, a point set of the number of times the API is called, the call success rate, and the average response time in the sub-window is determined; and the persistent homology feature of the point set is determined; each call data and each persistent homology feature are input into a pre-trained SVM recognition model to obtain the detection result of whether the API is abnormal output by the SVM recognition model. Because in the embodiment of the present application, the electronic device can accurately detect whether the API is abnormal without relying on predefined rules through API call data, deep feature extraction based on persistent homology, and a pre-trained SVM recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 A schematic diagram of a process of an anomaly detection method provided in an embodiment of the present application;
[0019] Figure 2 A detailed process diagram of anomaly detection provided in an embodiment of the present application;
[0020] Figure 3 A structural diagram of an abnormality detection device provided in an embodiment of the present application;
[0021] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.
[0023] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0025] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.
[0026] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0027] In order to accurately and effectively detect whether an API is abnormal, the embodiments of the present application provide an abnormality detection method, device, equipment and medium.
[0028] The anomaly detection method comprises: an electronic device obtains each call record of an API to be detected within a preset time period, wherein any call record contains a plurality of call data; wherein the call data includes a response time and a result of whether the call is successful; the preset time period is divided into sub-windows of preset time lengths, and for each sub-window, a point set of the number of times the API is called, the call success rate, and the average response time in the sub-window is determined; and a persistent homology feature of the point set is determined; each call data and each persistent homology feature are input into a pre-trained SVM recognition model, and a detection result of whether the API is abnormal output by the SVM recognition model is obtained.
[0029] Embodiment 1:
[0030] Figure 1 A schematic diagram of a process of an anomaly detection method provided in an embodiment of the present application, the process includes the following steps:
[0031] S101: Obtain each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful.
[0032] The anomaly detection method provided in the embodiment of the present application is applied to electronic devices, which include intelligent devices such as PCs or servers.
[0033] In order to accurately and effectively detect whether the API is abnormal, the electronic device can obtain each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data. The call data includes the response time and whether the call is successful.
[0034] The preset time period may be 1 minute.
[0035] In one possible implementation, the electronic device can collect complete call logs from the API server and obtain each call record of the API to be tested based on the call logs. Specifically, a unified log format standard is established, a detailed data collection strategy is formulated, and automated tools are deployed to capture data regularly. The collected logs are stored in a reliable distributed file system or cloud storage service, and strict data management policies are implemented, including periodic backup and access control. The log record data should include the following fields:
[0036] Timestamp: records the specific time of each API call;
[0037] API name: records the name or endpoint of the API being called;
[0038] Call result: record the response status of each API call, such as success (200OK), not found (404NotFound), server error (500Internal Server Error), etc.
[0039] Response time: records the response time of each API call, that is, the time from when the request is sent to when the response is received;
[0040] User information: records the information of the user or device that initiates the call, such as user ID, IP address, device type, etc.
[0041] S102: Divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine the persistent homology feature of the point set.
[0042] The electronic device can divide the preset time period into sub-windows of preset time length. Specifically, a time series call data set in seconds can be formed in chronological order, and the windows can be divided according to the minute dimension. The data in each window will represent all API call activities in the preset time period. There can be overlapping time periods in each sub-window. The preset time length of these sub-windows can be set to 5 seconds.
[0043] That is to say, each sub-window covers not only the current time point, but also the previous time points. This overlapping sub-window division method makes the data set in each sub-window contain not only the information of the current time point, but also the historical data of the previous time points, which helps to analyze the time series characteristics and patterns in the API call log more deeply.
[0044] For example, the time period of 0-5 seconds may be regarded as the first sub-window, followed by the time period of 1-6 seconds constituting the second sub-window, and so on.
[0045] For each sub-window, the electronic device can calculate and determine the number of times the API is called, the call success rate, and the point set of the average response time in the sub-window. Once the point set of the sub-window is obtained, the electronic device can use the persistent homology method to further analyze and extract the persistent homology features of the point set. Among them, the persistent homology method is a method for analyzing the topological structure of high-dimensional data. It captures the topological features of the data at different scales by calculating the persistent strips of the simplified complex. The persistent strip represents the survival time of the topological feature at different scales. The longer the persistent strip, the more important the corresponding topological feature.
[0046] It should be noted that local information is analyzed by creating smaller windows within a larger sliding window of a preset time period, and the statistical data of these sub-windows represent the features of the large window. This method can provide more detailed analysis without increasing the computational burden when processing time series data. Compared with directly calculating features on the entire long sequence, the sub-window method can process large-scale data more efficiently through parallel processing and local feature calculation. At the same time, extracting features within a small window can reduce the impact of noise in the long sequence and improve the stability of the features. Through the time series embedding technology, the original one-dimensional time series data is converted into a point set in a high-dimensional space, which enhances the expressive power of the data and enables the model to capture more complex patterns and relationships. At the same time, the optimized sub-window method allows the creation of smaller, overlapping sub-windows within a larger time window, which not only helps to capture local patterns and behaviors, but also optimizes the use of computing resources.
[0047] S103: Input each call data and each persistent homology feature into a pre-trained SVM recognition model, and obtain a detection result of whether the API is abnormal output by the SVM recognition model.
[0048] In order to perform anomaly detection accurately and effectively, the electronic device locally stores a pre-trained SVM recognition model. After obtaining the persistent homology features of each call data and the point set of each subwindow, the electronic device can input each call data and each persistent homology feature into the SVM model and obtain the output of the SVM model. The output of the SVM model is the detection result of whether the API is abnormal.
[0049] Extract basic statistical features from the API call data in each time window, count the total number of API calls in each time window, calculate the call success rate, that is, the ratio of the number of successful calls to the total number of calls, and calculate the average response time of all calls.
[0050] It should be noted that persistent homology technology combines data enhancement and machine learning (such as SVM model) to detect API anomalies. For advanced attack technologies such as low-frequency and slow attacks, it can identify these behaviors by analyzing the global and local topological structures of API calls, expand adaptive learning capabilities through SVM, automatically process real-time data streams, and quickly identify abnormal behaviors without human intervention. The features extracted from the persistent homology graph are combined with traditional statistical features to provide a more comprehensive API behavior representation and improve the performance of the classifier.
[0051] In an embodiment of the present application, when the electronic device is training the SVM recognition model. A label is assigned to each time window of a preset time period, and this label classifies the API call behavior as "normal" or "abnormal" based on historical data and behavior analysis. In an embodiment of the present application, a radial basis function (RBF) can be used as the kernel function of the SVM, and a cross grid search technique can be used to determine the optimal kernel coefficient γ and regularization parameter C. This process aims to find the parameters of the SVM recognition model that can best distinguish between normal and abnormal behaviors through an exhaustive parameter combination search. The electronic device can use the feature vector composed of the call data and the persistent homology feature and the assigned label as the training set data to train the SVM recognition model, learn the characteristic patterns of normal and abnormal behaviors, and predict whether the new API call behavior is abnormal.
[0052] In traditional technologies, with the diversification of API call patterns and the huge scale of log data, rule-based detection methods often need to be frequently updated and adjusted, with high maintenance costs and difficulty in adapting to rapidly changing environments. With the continuous development of machine learning and data mining technologies, researchers have begun to explore the use of these technologies for API anomaly detection. Machine learning methods can automatically identify anomalies without predefined rules by learning patterns of normal and abnormal behaviors from a large amount of log data. However, traditional machine learning methods mainly rely on statistical features of time series, such as mean, variance, frequency, etc. When processing complex time series data, these methods often fail to capture the deep structural information of the data, resulting in unsatisfactory detection results. The embodiment of the present application designs and proposes an API anomaly detection method based on persistent coherence technology. In order to improve the accuracy and adaptability of detection, a method that can identify abnormal behavior from complex API call patterns is required. The embodiment of the present application analyzes the time series data of API calls, and deep feature extraction based on persistent coherence can accurately identify API anomalies without relying on predefined rules. By aggregating the SVM recognition model and extracting advanced statistical features from the persistent homology graph and combining them with traditional statistical features, a comprehensive API call behavior feature vector is constructed, which can comprehensively consider the different properties and importance of the features, optimize the nonlinear relationship between the features, and further improve the accuracy of detection. The embodiment of the present application provides an automated, efficient, accurate and highly adaptable API anomaly detection solution by combining modern mathematical topology theory, data enhancement technology, machine learning algorithms and feature engineering. It can significantly improve the security and reliability of API services, reduce potential security risks and system failures, and protect the interests of enterprises and users.
[0053] In the embodiment of the present application, the electronic device deploys the trained SVM recognition model into the production environment for monitoring the real-time API call log. The real-time API call log is subjected to the same preprocessing and feature extraction steps, and the feature vector of the real-time data is predicted using the trained model to determine whether it is an abnormal behavior. When the model predicts abnormal behavior, an alarm mechanism is triggered.
[0054] In an embodiment of the present application, an electronic device obtains each call record of an API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful; the preset time period is divided into sub-windows of preset time length, and for each sub-window, a point set of the number of times the API is called, the call success rate, and the average response time in the sub-window is determined; and the persistent homology feature of the point set is determined; each call data and each persistent homology feature are input into a pre-trained SVM recognition model to obtain the detection result of whether the API is abnormal output by the SVM recognition model. Because in the embodiment of the present application, the electronic device can accurately detect whether the API is abnormal without relying on predefined rules through API call data, deep feature extraction based on persistent homology, and a pre-trained SVM recognition model.
[0055] Embodiment 2:
[0056] In order to accurately and effectively perform anomaly detection, based on the above embodiment, in the embodiment of the present application, after obtaining each call record of the API to be detected within a preset time period, and before determining the number of calls, call success rate, and average response time of the API in each sub-window, the method further includes:
[0057] For each acquired call record, if any preset type of call data does not exist in the call record, or the transfer parameters contained in the call record are not identified, the call record is deleted.
[0058] In actual scenarios, a call record of an API may be abnormal, and it is impossible to effectively detect whether the API is abnormal based on the call data in the call record. In order to accurately and effectively identify abnormalities, the electronic device can remove invalid or erroneous call records.
[0059] In order to accurately and effectively perform anomaly detection, the electronic device can determine whether any preset type of call data does not exist in each acquired call record. If any preset type of call data does not exist in the call record, it is impossible to effectively identify whether the API is abnormal based on the call data in the call record, so the call record can be deleted. The preset type can be a timestamp, API name, etc.
[0060] The call data in the call record also includes the passed parameters, which refer to the parameters passed each time the API is called. The electronic device identifies the passed parameters contained in the call record. If the electronic device does not identify the parameters, it cannot effectively identify whether the API is abnormal based on the call data in the call record, so the call record can be deleted.
[0061] Electronic devices can also eliminate records with obvious errors, such as illogical timestamps.
[0062] Embodiment 3:
[0063] In order to accurately and effectively perform anomaly detection, based on the above embodiment, in the embodiment of the present application, after obtaining each call record of the API to be detected within a preset time period, before inputting each call data and each persistent homology feature into the pre-trained SVM recognition model, the method further includes:
[0064] Get each call data in each call record;
[0065] For each of the calling data, a target format saved for the calling data is obtained, and the calling data is converted into data in the target format.
[0066] In order to ensure data consistency and thus accurately and effectively perform anomaly detection, electronic devices can perform standardized processing on the call data.
[0067] Specifically, the electronic device can obtain each call data in each call record. After obtaining each call data, the electronic device can obtain the target format saved for each call data and convert the call data into data in the target format.
[0068] The electronic device can convert the timestamp into a unified time format ISO 8601, convert all parameter names into lowercase, remove extra spaces or special characters, and arrange them in a predefined order. The timestamp is converted into a second-level time point to facilitate counting the number of API calls per second.
[0069] Embodiment 4:
[0070] In order to accurately and effectively perform anomaly detection, based on the above embodiment, in the embodiment of the present application, after determining the persistent homology feature of the point set, before inputting each call data and each persistent homology feature into the pre-trained SVM recognition model, the method further includes:
[0071] Each of the call data and each of the persistent coherent features are normalized.
[0072] In order to accurately and effectively perform anomaly detection, after obtaining the persistent homology features of the point set, the electronic device normalizes each call data and each persistent homology feature. Specifically, the electronic device can perform Min-Max normalization processing.
[0073] After obtaining the normalized call data and persistent homology features, the call data and the persistent homology features are combined in a preset order to form a complete feature vector, which is then input into the SVM model.
[0074] Embodiment 5:
[0075] In order to perform anomaly detection accurately and effectively, on the basis of the above embodiments, in an embodiment of the present application, the persistent coherence feature includes: a Betti curve feature in the persistent coherence feature.
[0076] The persistent coherent feature in the embodiment of the present application includes a Betti curve feature in the persistent coherent feature.
[0077] Specifically, the electronic device can first create an arithmetic sequence with a spacing of 5 as each domain parameter in the point set of each sub-window, traverse the neighborhood parameters for each sub-window point set, and calculate the distance matrix between the data points. Determine the Euclidean distance between each pair of points, and construct the Rips complex with different domain parameters as thresholds. If two data points are within this distance threshold, the two points are connected to each other to form an edge. As this domain parameter gradually increases, more points will be connected, gradually forming a complex topological structure. After the Rips complex is constructed, 0-dimensional homology features and 1-dimensional homology features are extracted. Based on the 0-dimensional homology features and the 1-dimensional homology features, the Betti curve features are calculated. The continuous Betti curve is discretized for easy use as a feature input. A series of neighborhood parameters are selected, the Betti number at each neighborhood parameter is calculated, and then each Betti number is formed into a vector as part of the feature vector input into the SVM recognition model.
[0078] Among them, the 0-dimensional homology feature represents the connected components. Each point is a separate connected component, that is, when there is no edge between the points, each point is an independent connected component. When the domain parameter increases to the point where some points begin to connect, the number of connected components of these points will decrease. By tracking the generation and merging process of connected components, the birth and death time of each connected component is recorded. The birth time refers to the parameter value when the point first appears when the domain parameter is small enough, and the death time refers to the parameter value when the domain parameter increases to the point where two connected components merge into one. The 1-dimensional homology feature represents the cyclic structure. When the domain parameter increases further, more points begin to connect and form a ring. The 1-dimensional homology feature records the generation and disappearance process of these rings. In order to find these rings in the program, the depth-first search (DFS) algorithm is used. The persistence of the ring is calculated by recording the time of its formation and extinction. The birth time refers to the parameter value when the domain parameter increases to the point where a ring is first formed, and the death time refers to the parameter value when the domain parameter increases to the point where the ring breaks.
[0079] Embodiment 6:
[0080] In order to accurately and effectively perform anomaly detection, based on the above embodiment, in the embodiment of the present application, after determining the persistent homology feature of the point set, before inputting each call data and each persistent homology feature into the pre-trained SVM recognition model, the method further includes:
[0081] Determine the persistent homology graph of each point set; calculate the Wasserstein distance feature and the bottleneck distance feature based on the persistent homology graph of every two point sets; determine the target Wasserstein distance feature and the target bottleneck distance feature for the preset time period according to the average value of each Wasserstein distance feature and the average value of each bottleneck distance feature obtained;
[0082] And for each point set; according to the 0-dimensional homology feature and 1-dimensional homology feature of the Rips complex of the point set, determine the persistence time of the point set;
[0083] Based on the persistence time of each point set, determining a persistence time feature and a persistence entropy; determining an average value of each persistence entropy;
[0084] The step of inputting each of the call data and each of the persistent homology features into a pre-trained SVM recognition model comprises:
[0085] Each of the call data, each persistent homology feature, the persistent time feature, the average value, the target Wasserstein distance feature and the target bottleneck distance feature are input into a pre-trained SVM recognition model.
[0086] In order to accurately and effectively perform anomaly detection, the electronic device can determine the persistent homology graph of each point set, and calculate the Wasserstein distance feature and the bottleneck distance feature based on the persistent homology graphs of every two point sets. These two distance measurement methods are used to compare the differences between two persistent homology graphs. Among them, the Wasserstein distance measures the overall difference between the two graphs, while the bottleneck distance measures the maximum difference between the two graphs. The distance calculation results between all sub-windows are averaged to generate the target Wasserstein distance and target bottleneck distance features for the preset time period.
[0087] And for each point set; according to the 0-dimensional homology feature and 1-dimensional homology feature of the Rips complex of the point set, determine the persistence time of the point set;
[0088] The average value of the persistence time of each point set is determined as the persistence time feature of the preset time period; the electronic device also calculates the persistence entropy according to the persistence time of each point set; and determines the average value of the persistence entropy of each point set.
[0089] Specifically, the electronic device can use the following formula to determine the persistent entropy:
[0090]
[0091] Among them, C is the feature set in the persistent homology graph, and Lc is the persistence time of feature c.
[0092] The specific method of calculating the persistent entropy is known in the art and will not be described in detail here.
[0093] Each call data, each persistent homology feature, persistent time feature, average value, target Wasserstein distance feature and target bottleneck distance feature are input into the pre-trained SVM recognition model.
[0094] It should be noted that traditional methods often perform poorly when processing high-dimensional complex data. This application effectively solves this problem by introducing persistent homology technology in topological data analysis, and provides a new method for extracting multi-dimensional features from time series data, enhancing the expressiveness of features. By constructing a topological graph of the time series and calculating its topological features, such as Betti numbers and persistent homology graphs, it is possible to capture the multi-scale topological features of the data, identify potential patterns in the data, provide richer feature information, and improve the accuracy and robustness of anomaly detection.
[0095] The electronic device can also extract basic statistical features from the API call data in each time window, count the total number of API calls in each time window, calculate the success rate of the calls, that is, the ratio of the number of successful calls to the total number of calls, calculate the average of all call response times, and input the basic statistical features into the SVM recognition model.
[0096] In an embodiment of the present application, the electronic device can select the embedding dimension d to embed the one-dimensional time series data into a higher-dimensional space to better capture the structure and characteristics of the data, and then create an embedding vector. For each time point, a vector containing the current and previous d-1 time point data is created. The one-dimensional time series is converted into a d-dimensional vector sequence, providing more dimensional information and structural features for subsequent analysis. After verification, the embedding dimension d=3 is selected, and the corresponding embedding vector corresponding to the time point t is [x(t), x(t-1), x(t-2)], where x(t)x represents the call data at time point t. The embodiment of the present application converts the API call sequence into a point set in a multidimensional space through the data enhancement method of the time delay embedding method. The continuous API call sequence can be converted into points in the state space to generate a feature matrix. Converting the time series data into a more complex topological structure in this way helps to extract richer homology features, which can not only improve the robustness of the model to small changes in data, but also effectively enhance the generalization ability of the model, making it more flexible and accurate when processing data in practical applications.
[0097] Figure 2 A detailed process diagram of anomaly detection provided in an embodiment of the present application.
[0098] Depend on Figure 2 It can be seen that the electronic device can first collect data samples, pre-process the data, format the data and divide the windows, perform basic feature statistics and feature data enhancement, perform sub-window division and time delay embedding based on the data after the sub-data enhancement, and perform persistent homology feature calculation, specifically, perform Rips complex construction and topological feature calculation;
[0099] After acquiring each feature, the feature is normalized, and the SVM recognition model is trained based on the normalized features; real-time anomaly detection is performed based on the trained SVM recognition model.
[0100] Embodiment 7:
[0101] In practice, the method provided in the embodiments of the present application can be implemented and applied in multiple scenarios, as listed below:
[0102] 1. Cloud service API monitoring
[0103] In a cloud service environment, the stability and security of the API are crucial. The method proposed in the embodiment of the present application can be applied to the API monitoring system of a cloud service provider. By performing persistent coherence analysis on the API call log, abnormal call patterns caused by malicious attacks or system failures can be detected in real time. For example, when a sudden increase in the call frequency of an API or an abnormal parameter value is detected, the system will automatically trigger an alarm and take corresponding defensive measures, such as limiting the request frequency or temporarily blocking the IP address of the device with the abnormal call. In addition, the application of data enhancement technology enables the model to adapt to different attack methods and improves the robustness of the detection system. By implementing the method of the embodiment of the present application, cloud service providers can ensure the security and reliability of their API services and enhance user trust.
[0104] 2. Financial Services API Security
[0105] In the financial services industry, API security is the key to ensuring transaction security and customer data privacy. The method proposed in the embodiment of the present application can be applied to API security monitoring of banks or securities trading platforms. By performing persistent coherence analysis on the call data of the transaction API, abnormal transaction patterns can be identified. For example, when the system detects that an account has initiated a large number of high-frequency transaction requests in a short period of time and there are abnormalities in the transaction parameters, it can be determined that the account may have been subjected to a fraud attack. At this point, the system will automatically restrict the transaction behavior of the account and notify the risk management department for further review. By implementing the method of the embodiment of the present application, financial service institutions can promptly detect and prevent potential security threats and protect the safety of customer assets.
[0106] 3. Enterprise internal API management
[0107] In large enterprises, internal APIs are frequently called, involving multiple departments and systems. The method proposed in the embodiment of the present application can be applied to the monitoring and management of internal APIs in enterprises. By performing persistent coherence analysis on the call data of internal APIs, abnormal behaviors caused by system failures, configuration errors, or internal security threats can be effectively identified. For example, when a department's API call suddenly exceeds the normal range, or an abnormal value appears in the call parameter, the system will automatically issue a warning and notify the IT department to investigate. In addition, the application of data enhancement technology enables the model to adapt to different usage scenarios and data distributions, improving the accuracy and adaptability of the detection system. By implementing the method of the embodiment of the present application, enterprises can ensure the stability and security of internal APIs and reduce system failures and security risks.
[0108] Embodiment 8:
[0109] Based on the same inventive concept, the present application embodiment also provides an abnormality detection device, please refer to Figure 3, the device comprises:
[0110] The acquisition module 301 is used to acquire each call record of the application programming interface API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes a response time and a result of whether the call is successful;
[0111] The processing module 302 is used to divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine the persistent homology features of the point set; input each call data and each persistent homology feature into a pre-trained support vector machine (SVM) recognition model, and obtain a detection result of whether the API is abnormal output by the SVM recognition model.
[0112] In a possible implementation, the processing module 302 is further configured to delete each acquired call record if any preset type of call data does not exist in the call record, or if a transfer parameter contained in the call record is not identified.
[0113] In a possible implementation, the processing module 302 is further configured to obtain each call data in each call record; for each call data, obtain a target format saved for the call data, and convert the call data into data in the target format.
[0114] In a possible implementation, the processing module 302 is further configured to normalize each call data and each persistent homology feature.
[0115] In a possible implementation, the processing module 302 is further used to determine the persistent homology graph of each point set; based on the persistent homology graphs of every two point sets, calculate the Wasserstein distance feature and the bottleneck distance feature; determine the target Wasserstein distance feature and the target bottleneck distance feature of the preset time period according to the obtained average value of each Wasserstein distance feature and the average value of each bottleneck distance feature; and for each point set; determine the persistence time of the point set according to the 0-dimensional homology feature and the 1-dimensional homology feature of the Rips complex of the point set; determine the persistence time feature and the persistence entropy based on the persistence time of each point set; determine the average value of each persistence entropy;
[0116] The processing module 302 is specifically used to input each call data, each persistent homology feature, the persistent time feature, the average value, the target Wasserstein distance feature and the target bottleneck distance feature into a pre-trained SVM recognition model.
[0117] Embodiment 9:
[0118] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the function of abnormality detection mentioned above, please refer to Figure 4 The device includes a processor 401, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.
[0119] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:
[0120] Obtain each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful;
[0121] Divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine a persistent homophonic feature of the point set;
[0122] Each call data and each persistent homology feature are input into a pre-trained SVM recognition model, and a detection result of whether the API is abnormal output by the SVM recognition model is obtained.
[0123] In a possible implementation, after acquiring each call record of the API to be detected within a preset time period and before determining, for each sub-window, a point set of the number of calls, call success rate, and average response time of the API in the sub-window, the method further includes:
[0124] For each acquired call record, if any preset type of call data does not exist in the call record, or the transfer parameters contained in the call record are not identified, the call record is deleted.
[0125] In a possible implementation, after acquiring each call record of the API to be detected within a preset time period and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0126] Get each call data in each call record;
[0127] For each of the calling data, a target format saved for the calling data is obtained, and the calling data is converted into data in the target format.
[0128] In a possible implementation, after determining the persistent homology features of the point set and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0129] Each of the call data and each of the persistent coherent features are normalized.
[0130] In a possible implementation, the persistent coherent feature includes: a Betti curve feature in the persistent coherent feature.
[0131] In a possible implementation, after determining the persistent homology features of the point set and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0132] Determine the persistent homology graph of each point set; calculate the Wasserstein distance feature and the bottleneck distance feature based on the persistent homology graph of every two point sets; determine the target Wasserstein distance feature and the target bottleneck distance feature for the preset time period according to the average value of each Wasserstein distance feature and the average value of each bottleneck distance feature obtained;
[0133] And for each point set; according to the 0-dimensional homology feature and 1-dimensional homology feature of the Rips complex of the point set, determine the persistence time of the point set;
[0134] Based on the persistence time of each point set, determining a persistence time feature and a persistence entropy; determining an average value of each persistence entropy;
[0135] The step of inputting each of the call data and each of the persistent homology features into a pre-trained SVM recognition model comprises:
[0136] Each of the call data, each persistent homology feature, the persistent time feature, the average value, the target Wasserstein distance feature and the target bottleneck distance feature are input into a pre-trained SVM recognition model.
[0137] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0138] The communication interface 402 is used for communication between the electronic device and other devices.
[0139] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0140] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
[0141] Embodiment 10:
[0142] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:
[0143] Obtain each call record of the API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes response time and whether the call is successful;
[0144] Divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine a persistent homophonic feature of the point set;
[0145] Each call data and each persistent homology feature are input into a pre-trained SVM recognition model, and a detection result of whether the API is abnormal output by the SVM recognition model is obtained.
[0146] In a possible implementation, after acquiring each call record of the API to be detected within a preset time period and before determining, for each sub-window, a point set of the number of calls, call success rate, and average response time of the API in the sub-window, the method further includes:
[0147] For each acquired call record, if any preset type of call data does not exist in the call record, or the transfer parameters contained in the call record are not identified, the call record is deleted.
[0148] In a possible implementation, after acquiring each call record of the API to be detected within a preset time period and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0149] Get each call data in each call record;
[0150] For each of the calling data, a target format saved for the calling data is obtained, and the calling data is converted into data in the target format.
[0151] In a possible implementation, after determining the persistent homology features of the point set and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0152] Each of the call data and each of the persistent coherent features are normalized.
[0153] In a possible implementation, the persistent coherent feature includes: a Betti curve feature in the persistent coherent feature.
[0154] In a possible implementation, after determining the persistent homology features of the point set and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes:
[0155] Determine the persistent homology graph of each point set; calculate the Wasserstein distance feature and the bottleneck distance feature based on the persistent homology graph of every two point sets; determine the target Wasserstein distance feature and the target bottleneck distance feature for the preset time period according to the average value of each Wasserstein distance feature and the average value of each bottleneck distance feature obtained;
[0156] And for each point set; according to the 0-dimensional homology feature and 1-dimensional homology feature of the Rips complex of the point set, determine the persistence time of the point set;
[0157] Based on the persistence time of each point set, determining a persistence time feature and a persistence entropy; determining an average value of each persistence entropy;
[0158] The step of inputting each of the call data and each of the persistent homology features into a pre-trained SVM recognition model comprises:
[0159] Each of the call data, each persistent homology feature, the persistent time feature, the average value, the target Wasserstein distance feature and the target bottleneck distance feature are input into a pre-trained SVM recognition model.
[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0164] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An anomaly detection method, characterized in that: The method comprises: Obtain each call record of the application programming interface API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes a response time and a result of whether the call is successful; Divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine a persistent homophonic feature of the point set; Each call data and each persistent homology feature are input into a pre-trained support vector machine (SVM) recognition model, and a detection result of whether the API is abnormal output by the SVM recognition model is obtained.
2. The method according to claim 1, characterized in that After acquiring each call record of the API to be detected within a preset time period, and before determining, for each sub-window, a point set of the number of calls, call success rate, and average response time of the API in the sub-window, the method further includes: For each acquired call record, if any preset type of call data does not exist in the call record, or the transfer parameters contained in the call record are not identified, the call record is deleted.
3. The method according to claim 1, characterized in that After obtaining each call record of the API to be detected within a preset time period, and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes: Get each call data in each call record; For each of the calling data, a target format saved for the calling data is obtained, and the calling data is converted into data in the target format.
4. The method according to claim 1, characterized in that: After determining the persistent homology features of the point set and before inputting each call data and each persistent homology feature into the pre-trained SVM recognition model, the method further includes: Each of the call data and each of the persistent coherent features are normalized.
5. The method according to claim 1, characterized in that The persistent coherence feature includes: a Betti curve feature in the persistent coherence feature.
6. The method according to claim 1, characterized in that After determining the persistent homology features of the point set, and before inputting each call data and each persistent homology feature into a pre-trained SVM recognition model, the method further includes: Determine the persistent homology graph of each point set; calculate the Wasserstein distance feature and the bottleneck distance feature based on the persistent homology graph of every two point sets; determine the target Wasserstein distance feature and the target bottleneck distance feature for the preset time period according to the obtained average value of each Wasserstein distance feature and the average value of each bottleneck distance feature; And for each point set; according to the 0-dimensional homology feature and 1-dimensional homology feature of the Rips complex of the point set, determine the persistence time of the point set; Based on the persistence time of each point set, determining a persistence time feature and a persistence entropy; determining an average value of each persistence entropy; The step of inputting each of the call data and each of the persistent homology features into a pre-trained SVM recognition model comprises: Each of the call data, each persistent homology feature, the persistent time feature, the average value, the target Wasserstein distance feature and the target bottleneck distance feature are input into a pre-trained SVM recognition model.
7. An abnormality detection device, characterized in that: The device comprises: An acquisition module is used to acquire each call record of the application programming interface API to be detected within a preset time period, wherein any call record contains multiple call data; wherein the call data includes a response time and a result of whether the call is successful; A processing module is used to divide the preset time period into sub-windows of preset time lengths, and for each sub-window, determine a point set of the number of calls, call success rate, and average response time of the API in the sub-window; and determine the persistent homology features of the point set; input each call data and each persistent homology feature into a pre-trained support vector machine (SVM) recognition model, and obtain a detection result of whether the API is abnormal output by the SVM recognition model.
8. The device according to claim 7, characterized in that The processing module is further configured to delete each acquired call record if any preset type of call data does not exist in the call record, or if a transfer parameter contained in the call record is not identified.
9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for abnormality detection in any one of claims 1 to 6 according to the obtained program instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method for abnormality detection according to any one of claims 1 to 6.