Method, system and equipment for monitoring abnormal calling of API gateway of big data platform

Abstract

The invention discloses a method, a system and equipment for monitoring abnormal calling of an API gateway of a big data platform. The method comprises the following steps: acquiring log data and performance index data called by the API gateway of the big data platform; performing feature screening on feature fields of the log data through an Xgboost algorithm, and constructing a new input data set and a feature set in combination with performance index data; establishing an anomaly recognition model based on the improved double-layer random forest model; and deploying the established exception identification model at an API gateway side, capturing a call log, an IP and related performance data online, inputting the exception identification model, and completing exception call identification at the gateway side. According to the method and the system, online reasoning service with fast abnormal calling and high accuracy can be realized, potential safety hazards in a big data platform can be accurately identified, corresponding fusing operation can be conveniently carried out, and current limiting can be carried out on suspected dangerous users in time.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal gateway call monitoring, and more specifically to a method, system and device for monitoring abnormal calls of a big data platform API gateway. Background Art

[0002] At present, with the continuous updating of technology, the close integration of big data platform and microservice architecture has made the platform more open while carrying service functions and tenant calls. The platform's call security and platform operation and maintenance are facing greater challenges.

[0003] In the daily operation and maintenance of mainstream big data platforms, there have always been problems such as frequent user access, suspected active attacks, and abnormal problems in the platform interface itself, which threaten the security and operational stability of the platform. These problems can be roughly divided into two types: system call abnormalities and access abnormalities. Currently, there are usually two methods for identifying abnormalities in API gateway calls, both of which have certain defects.

[0004] The first is the mainstream solution for API gateway calls on big data platforms. Based on the historical experience of business experts, the upper threshold of the platform access capacity is set to set business rules. The solutions to abnormal problems are mainly based on circuit breakers and current limiting strategies. This method detects the fluctuations of the main performance indicators of the big data platform, and does not deeply explore the causes of the abnormalities. There are certain security risks, and the subsequent investigation is time-consuming and labor-intensive. At the same time, the disadvantage of this approach is that traditional abnormal calls are identified by setting rules, which has problems such as unreasonable rule settings and long rule update cycles. Since the rules themselves are set manually and the rules are generated by general business logic, and the sources of abnormal calls are diverse and variable, the abnormal call identification method summarized by general rules may lead to missed abnormal identification, and normal calls may also be intercepted, which leads to low accuracy in abnormal call identification and low actual use value of the function.

[0005] The second method is mainly to model or logically judge the log information of the big data platform, encode and classify the log information obtained in real time, and filter out abnormal logs through a large natural language processing model. Compared with the first method, modeling for log information can expose the security risks of the big data platform in a timely manner, reduce the risk of being attacked, and increase the stability of the platform. However, this method still ignores the timeliness of the abnormal call detection requirements, and similar to the log information anomaly detection modeling methods such as deep neural networks, there is a problem of difficulty in extracting model features. The model itself is highly complex, and the network with better fitting effect is relatively large in size and slow in running time. In the scenario where the API gateway interface is frequently called, the use efficiency is low. Moreover, this approach ignores the fluctuation of KPI performance indicators caused by abnormal calls. The focus of modeling is to determine the source of abnormalities by log tracing, but ignores the specific type of abnormal calls, and the model has low interpretability.

[0006] In general, the existing solutions for the problem of abnormal call identification of API gateways on big data platforms have the disadvantages of low accuracy and failure to guarantee high timeliness requirements. Therefore, it is urgent to solve the existing problems of accuracy and efficiency in abnormal call identification, increase the stability of API gateway operation, prevent network attacks, and realize fast and high-accuracy online reasoning services for abnormal calls. Summary of the invention

[0007] In view of this, the present invention provides a method, system and device for monitoring abnormal calls of API gateways of a big data platform that at least solve some of the above technical problems. The present invention uses the Xgboost algorithm to perform feature screening on the feature fields of log information, and combines API-related resource performance indicators to generate new input data sets and feature sets. Then, a high-fitting double-layer random forest model is used to establish an abnormality recognition model, and finally an online reasoning service with fast abnormal calls and high accuracy is realized through the abnormality recognition model.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] In a first aspect, the present invention provides a method for monitoring abnormal calls of a big data platform API gateway, the method comprising the following steps:

[0010] S1. Obtain the log data and performance indicator data of the API gateway call of the big data platform;

[0011] S2. Use the Xgboost algorithm to perform feature screening on the feature fields of the log data, and combine the performance indicator data to construct a new input data set and feature set;

[0012] S3. Use the data set and feature set to train the improved two-layer random forest model and establish an anomaly recognition model;

[0013] S4. Deploy the established exception recognition model on the API gateway side, capture call logs, IP and related performance data online, input the exception recognition model, and complete abnormal call recognition on the gateway side.

[0014] Furthermore, in S1, the obtained log data includes: access time, http access path, returned status code information and method name; each log sample generates specific fields with the user's calling address and calling time as a record, and at the same time, the second-level performance indicator data when each log occurs is associated with the log feature through time features; the performance indicator data includes: CPU utilization, second-level concurrency and server response time.

[0015] Furthermore, in S2, feature screening is performed based on the Xgboost algorithm with the structural risk term added, and the objective function of the Xgboost algorithm with the structural risk term added is:

[0016]

[0017]

[0018] Among them, g(x) represents the objective function, y i represents the true value, represents the predicted value, i represents the i-th sample, n represents the number of samples, L represents the loss function, fk represents the structural risk term, K represents the number of trees, T represents the number of leaf nodes, w represents the weight of the node, and a and b are hyperparameters.

[0019] Furthermore, in S3, the process of establishing an anomaly recognition model includes:

[0020] 1) Extract K training data sets from the data set with replacement. The number of samples in each training subset is N. These samples are called bootstrap samples to train the decision trees respectively.

[0021] 2) Use cart classification and regression tree to construct a decision tree. At the node of the tree, randomly select m features from M input features as the split feature set of the current node of the decision tree, select the optimal split feature and split point, and assign the training data set to the child node;

[0022] 3) Train K bootstrap sample sets in sequence and obtain all decision trees according to the set parameters;

[0023] 4) Prune the forest, construct a two-layer random forest model, select the optimal subtree, select the top N abnormal categories with probability of the cart decision tree, and introduce counter i=1 when the predicted value output by the model is equal to the true value, that is, it returns correct. When the true label is not equal to the predicted label, counter i=0; by constructing the loss function, find out several decision trees whose loss function is less than the target value to form a sub-forest.

[0024] Furthermore, the Gini coefficient minimization criterion is used to select the optimal splitting features and splitting points.

[0025] Furthermore, the performance of the anomaly recognition model was verified through precision, recall, PR curve, F1 score, ROC curve and AUC value.

[0026] In a second aspect, the present invention further provides a system for monitoring abnormal calls of a big data platform API gateway, which is applied to the above-mentioned method for monitoring abnormal calls of a big data platform API gateway to realize abnormal call identification, and the system includes:

[0027] The data acquisition module is used to obtain the log data and performance indicator data of the API gateway call of the big data platform;

[0028] The feature screening and data set construction module is used to screen the feature fields of log data through the Xgboost algorithm, and construct a new input data set and feature set in combination with performance indicator data;

[0029] The anomaly recognition model building module is used to train the improved two-layer random forest model using the data set and feature set to build an anomaly recognition model;

[0030] The abnormal call identification module is used to deploy the established abnormal call identification model on the API gateway side, capture call logs, IP and related performance data online, input the abnormal call identification model, and complete abnormal call identification on the gateway side.

[0031] In a third aspect, the present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned method for monitoring abnormal calls of a big data platform API gateway.

[0032] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0033] The present invention provides a method, system and device for monitoring abnormal calls of API gateways on a big data platform. The present invention uses the Xgboost algorithm to perform feature screening on the feature fields of log information, and combines API-related resource performance indicators to generate new input data sets and feature sets. Then, a high-fitting double-layer random forest model is used to establish an abnormality recognition model, and finally an online reasoning service with fast abnormal calls and high accuracy is realized through the abnormality recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0035] Figure 1 A flowchart of a method for monitoring abnormal calls to an API gateway of a big data platform provided in an embodiment of the present invention.

[0036] Figure 2 A schematic diagram of an abnormal call identification architecture for a big data platform interface provided in an embodiment of the present invention.

[0037] Figure 3 A schematic diagram of the model training establishment process provided in an embodiment of the present invention.

[0038] Figure 4 A schematic diagram of the model application process provided by an embodiment of the present invention.

[0039] Figure 5 A schematic diagram of an anomaly recognition model established by a high-fitting two-layer random forest model provided in an embodiment of the present invention.

[0040] Figure 6 A schematic diagram of a deployment architecture for identifying abnormal calls to an API gateway on a big data platform provided in an embodiment of the present invention.

[0041] Figure 7 A schematic diagram of the system structure for monitoring abnormal calls of an API gateway of a big data platform provided in an embodiment of the present invention.

[0042] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0044] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0045] See also Figures 1 to 6 As shown, in order to solve the existing problems of accuracy and efficiency in identifying abnormal calls, increase the stability of API gateway operation, and prevent network attacks, an embodiment of the present invention proposes an improved double-layer random forest method for identifying abnormal calls, and constructs a high-fit double-layer random forest model to achieve fast and accurate identification of abnormal calls of API gateway. First, the feature fields of the log information are screened by the Xgboost algorithm, and a new input data set and feature set are generated in combination with API-related resource performance indicators. Then, a high-fit double-layer random forest model is used to establish an abnormal recognition model, and finally an improved machine learning model is used to achieve fast and high-accuracy online reasoning services for abnormal calls, ensuring the stable operation of the big data platform.

[0046] The principles and specific implementation methods involved in the method of the present invention are described in detail below:

[0047] 1. Principles of the method:

[0048] 1.1Xgboost algorithm:

[0049] The Xgboost algorithm belongs to the boosting framework. Therefore, the overall idea of ​​Xgboost is consistent with the overall idea of ​​the boosting framework:

[0050] The first step is to initialize f(predictor);

[0051] The second step is to adjust the residual tree H1 (through different decomposition criteria);

[0052] The third step is to update F1=F+H1;

[0053] The fourth step is to repeat the previous content until the model meets the convergence conditions. One of the highlights of Xgboost is to define a splitting criterion so that the loss of each optimal split point can be minimized compared to the loss of not splitting. This is one of the reasons why Xgboost is effective. Therefore, there is a method: define the structure score before the split minus the structure score after the split as the gain of the split score, and then find the split score with the largest win as the best split point. The meaning of the structure score is the minimum value of the loss function when the structure of the tree is known. The gain definition of Xgboost is to subtract the structure score before the split from the structure score after the split, and select the split point with the highest win rate as the best split point. Its importance lies in determining the split point, that is, the loss of the model after the split is the largest than before the split. The effect of fitting the residual tree with this gain definition method is very good.

[0054] 1.2 Improved Random Forest Algorithm:

[0055] Random forest refers to a manager that uses several trees to train and predict examples. Its score is obtained by voting by a group of numbers. Usually we use every tree in the forest. Random construction of the following algorithms:

[0056] In the first stage, n represents the number of learning files (samples) and M represents the number of special files.

[0057] In the second stage, m attributes are input to decide a node on the tree, where m should be very smaller than M.

[0058] The third point is that n points in n training files are returned by the sample to build a set of training (ie initial samples), and predict and evaluate its error from the non-selected files, using it.

[0059] The fourth point is to randomly select attributes of m for each node and decide each node in the tree based on these attributes. Based on these features, the best separation mode is calculated.

[0060] Random forests can generate accurate systems that can meet many input features and can evaluate the most important features at a time specified by the group. When a forest is built, it may produce unfair evaluations of common errors within it. If the data sets are not equal, the errors can be corrected. The improved random forest algorithm is to "prune" the sub-forests of the random forest, retain the optimal sub-tree group with high precision to make the forest smaller, so as to better classify abnormal or unknown accesses. The improved random forest reduces the number of sub-trees and integrates the best tree classifiers to make the model lighter. In essence, the improved random forest algorithm is essentially an integrated algorithm for learning fusion. While maintaining the original voting method, it overcomes the problem of overfitting of decision trees. For unbalanced samples, it usually has good tolerance to noise and externalities and has high prediction accuracy. It is widely used in various fields, including the abnormal call detection involved in the present invention, which is also a classification model.

[0061] 1.3 Evaluation indicators:

[0062] The model's validation indicators include precision, recall, PR curve, F1score, ROC curve, AUC value, etc.

[0063] The precision rate indicates the proportion of all predicted positive samples to all predicted positive samples;

[0064] The recall rate indicates the proportion of all predicted positive samples to all positive samples;

[0065] The PR curve is used to compare the goodness of fit of different models;

[0066] F1score is the harmonic mean of precision and recall;

[0067] The ROC curve can be used to evaluate the performance of the model. Each point on the ROC curve corresponds to a threshold. For a classifier, there will be a TPR and FPR under each threshold. The closer to 1, the better the model performance.

[0068] Under normal circumstances, the above indicators can also be used to compare the performance of different models. The present invention mainly compares the models established by the random forest algorithm and the improved algorithm through indicators such as precision, recall and F1-score. For the pruned random forest improved model to be established by the present invention, for a single subtree, by evaluating the loss function of the subtree, the subtrees are sorted according to the value of the loss function, and the top 50% of the best subtrees are selected to form the forest. The original voting mechanism is retained, and on the basis of integrated learning, the model is simplified. On the premise of ensuring high accuracy, the reasoning efficiency of the model is improved as much as possible, and the practicality of the model is improved.

[0069] 1.4 Abnormal call data and characteristics:

[0070] 1.4.1 API Gateway Log Information Feature Extraction:

[0071] Sort out the types and causes of abnormal calls, and explore the log features required by the model. In an embodiment of the present invention, the log features required by the model are divided into three categories: basic information features, HTTP field features, and time features. The basic information features are directly obtained from the extracted data; the HTTP field features need to be separated from the HTTP field and divided into two parts: path features and parameter features, which mainly include some length, number, and keyword feature fields; finally, the time features need to be separated from the time field, which mainly include some fields such as year, month, day, and working day. All the fields obtained by the specific log are shown in Table 1 below:

[0072] Table 1 Log information characteristics

[0073]

[0074] 1.4.2 KPI feature extraction:

[0075] In the embodiment of the present invention, the performance resource data of the API gateway and the time series of the access log at the corresponding time are used as the input of the model, and the KPI characteristics of the log occurrence period are combined to build an abnormal call identification algorithm. The performance resource characteristics include key performance indicators such as CPU utilization, second-level concurrency, and server response time that can reflect the call frequency. As shown in Table 2, it mainly includes user perception fields, server load fields, etc.

[0076] Table 2 Performance resource feature extraction

[0077]

[0078] 2. Model establishment:

[0079] The present invention uses an improved high-precision random forest algorithm model to detect abnormal access behaviors, builds a performance resource detection algorithm based on the system and resource structure of the API gateway, detects abnormal calls in real time through abnormal fluctuations in performance and resource indicators, and issues timely alarms. Use key performance indicators that can reflect the frequency of calls, such as CPU utilization, second-level concurrency, and server response time. The improved random forest algorithm model proposed by the performance resource detection model is based on the original algorithm. By retaining the algorithm structure of the similarity between the decision tree with high classification accuracy and the improved tree, it adds a structural layer for selecting high-precision sub-forests before the decision tree integration decision, thereby improving the efficiency of abnormal behavior detection. Big data platform interface abnormal call recognition architecture such as Figure 2 shown.

[0080] 2.1 Obtain log and performance indicator data:

[0081] Obtain data source information, obtain and parse the log information of the big data platform service call, obtain the server response log and parse it to obtain a structured solution. The original log information contains access time, http access path, returned status code information and method name, etc. Each sample generates a specific field userIp as a record with the user's call address and call time. It is recorded as sequence I = {x1, x2..., xn}. In an embodiment of the present invention, the log feature structure and performance indicator feature structure are shown in Table 3 and Table 4 respectively:

[0082] Table 3 Log feature structure

[0083]

[0084] Table 4 KPI characteristic structure

[0085]

[0086] Furthermore, the second-level performance resource data of each log occurrence is associated with the log features described in Table 3 through the time feature. After association, the user-level access samples are obtained. The original performance resource indicators mainly include CPU utilization, second-level concurrency and server response time, which are key performance indicators that can reflect the call frequency. The access time time, historical request number reqCou, current second-level request concurrency reqEru, single IP request number IPreq, and memory usage rateC are recorded as a sequence J = {y1, y2, ... yn}. By associating the occurrence time, a new field req_id is formed with the userlp access time and number, and the classification reasons for abnormal access are marked, and the sample label is recorded as R = {R1, R2, ... Rn}, where Rn is the label of the nth sample. The data set D is formed, where the feature field sequence is T = {I, J}.

[0087] 2.2 Feature screening of log information fields and data set construction based on GBDT algorithms:

[0088] 2.2.1 Feature screening based on the Xgboost algorithm with structural risk terms added:

[0089] The present invention solves the problem of multivariate screening under the same data structure by improving the loss function of Xgboost and adding the structural risk term of the tree. In this way, the growth structure of the tree will be constrained in the process of building the tree, reducing the overfitting problem. In this way, the objective function becomes:

[0090]

[0091] Among them, g(x) represents the objective function, y i is the actual value, is the observed value, i represents the i-th sample, n represents the number of samples, fk represents the structural risk term, and K represents the number of trees. The number of leaf nodes T and the weight of the node w represent the complexity of a tree. Where a and b are hyperparameters, so the objective function of the Xgboost algorithm with structural risk term becomes:

[0092]

[0093] Furthermore, the samples are sorted by the size of the eigenvalues, the loss difference under the split point is calculated, and the corresponding feature and split point when the loss difference is the largest are found, and this is used as the current node for splitting. The features are extracted according to the first quarter and third degree. At this point, the variable sequence I = {x1,x 2, …x n}, become I ` ={x1,x2,…,x 75％*n}.

[0094] 2.2.2 Construction of training data set:

[0095] Unlike the low interpretability of neural network parameter weights, the inherent interpretability of the decision tree used by Xgboost reduces the computational complexity of the algorithm and improves the interpretability of the entire model. The higher the importance score, the more important the feature indicator is and the greater its contribution to the dataset.

[0096] When the total number of splits of each tree is larger, the feature is better, I ` ={x1,x2,…,x 75％*n}, establish a new feature set T1 and the corresponding data set D1, T1 = {I ` ,J}.

[0097] 2.3 Construct a high-fitting double-layer random forest model to establish an anomaly recognition model:

[0098] 2.3.1 Overall structure of anomaly recognition model:

[0099] like Figure 3 and Figure 4 As shown, the present invention proposes a high-fitting double-layer random forest model to establish an abnormal recognition model construction process and application, mainly by obtaining logs and performance data information, further screening features through the Xgboost algorithm, and obtaining the final model based on the random forest algorithm of pruning. The model application process completes abnormal call identification on the gateway side by inputting logs and performance information.

[0100] 2.3.2 Build a double-layer random forest model with high fitting degree to establish an anomaly recognition model:

[0101] As Figure 5 shown, by constructing a pruned random forest, an abnormal call access model for the API gateway of the big data platform is obtained. The model input is the generated label R = {R i}, i = 1, 2…n, and the feature sets I 、 = {x i}, i = 1, 2, n, J = {y i}, j = 1, 2,…, n.

[0102] The specific design steps are as follows:

[0103] 1) Randomly draw K training data sets from the data set with replacement. For each training subset, their sample size is also N. These samples are called bootstrap samples and are used to train decision trees respectively.

[0104] 2) Select the cart classification and regression tree to construct the decision tree. At the node of the tree, randomly select m features (m < M) from M input features as the splitting feature set of the current node of the decision tree. Select the optimal splitting feature and splitting point, and allocate the training data set to two child nodes. The criterion for selecting the splitting feature and splitting point is the minimum Gini coefficient criterion. Repeat the partitioning process until the stopping condition is met and the number of samples in the node is less than the predetermined threshold.

[0105] 3) Train K bootstrap sample sets in turn, and obtain all decision trees according to the set parameters. And record the results of the test set as T = {T(x) i}, where i = 1, 2,…n.

[0106] 4) Next, prune the forest to construct a double-layer random forest model. Select the optimal subtree, and select the top N abnormal categories with the highest probability of the constructed cart decision tree. When the predicted value output by the model is equal to the true value, that is, when it returns correctly, introduce the counter i = 1. When the true label is not equal to the predicted label, the counter i = 0. By constructing a loss function, find the decision trees with small loss functions to form a sub-forest, and select the top several subtrees to form a new forest.

[0107] The second-layer random forest undertakes the screening work of the optimal subtree, ensuring the simplicity and lightweight of the model and eliminating redundant information. On the premise of ensuring the accuracy of the improved random forest output, it meets the real-time requirements of the big data platform operation and maintenance.

[0108] 2.4 Online fast and highly accurate inference service:

[0109] See Figure 6 As shown in the figure, it shows the deployment architecture of abnormal call identification of the big data platform API gateway. By deploying it on the big data platform API gateway side instead of directly on the service side, the real-time nature of the call model and the controllability of the returned results are guaranteed. When the execution end sends a call command, a single user flow limiting strategy with a dynamic value can be directly adopted to ensure the stable operation of the platform in the event of suspected security issues.

[0110] 3. Experimental verification:

[0111] 3.1 Experimental design:

[0112] The test configuration environment performed by the present invention is shown in Table 5:

[0113] Table 5 Test configuration environment

[0114]

[0115] The experimental data set in the embodiment of the present invention is selected as the historical usage data of a high-performance API gateway of a big data platform, the time series of KPI performance indicators, with a granularity of seconds, and the log information range is selected as the data for the whole year of 2023 as the training sample, and the data for a certain month in 2024 as the test data set.

[0116] This experiment uses a single-layer random forest and an improved double-layer random forest as the control group. The decision tree generation algorithm selected is the cart classification tree as the base, and the random subspace uses the log2M of the number of features + 1 as the random value. At the same time, the minimization criterion of the Gini coefficient is selected for feature selection, and the random index is set to 0.1. No pruning operation is performed, and the number of original trees is 100. During the growth process of fitting, it is not affected by pruning to ensure structural stability. The maximum depth of the tree is set to 10 according to the default value, and the voting method is used, and the classification of the majority of trees is used as the final prediction result.

[0117] 3.2 Experimental results:

[0118] The sample selected is the historical usage data of the high-performance API gateway of the big data platform. The improved model is recorded as fDRF(x) and compared with the traditional random forest fSRF(x) model. The precision and recall rates of the output results of the two models are compared. The confusion matrix of the model comparison is shown in Table 6 below.

[0119] Table 6 Comparison of experimental accuracy of different algorithms

[0120]

[0121] As shown in Table 6, the pruned random forest algorithm model has a greater improvement in precision while ensuring the recall rate compared to the traditional algorithm. This shows that after pruning the original forest and discarding the low-precision subtrees, the model has a greater improvement.

[0122] At the same time, compared with the use of decision tree algorithms, neural network algorithms, etc., these types of algorithms do not have the timeliness of abnormal call detection needs. The deep neural network through simple logs has many features that are difficult to extract, and the model itself is complex. The network with better fitting effect is relatively large and has a slow running time. The present invention fits the improved random forest model, that is, the double-layer high-fitting degree model, so that the model structure is lightweight and the results are highly accurate, which is advanced.

[0123] From the description of the above embodiments, those skilled in the art can know that the present invention provides a method for monitoring abnormal calls of a big data platform API gateway, which has the following advantages:

[0124] The present invention establishes a complete set of API call exception identification solutions on the API gateway side by building a complete closed-loop process. Field screening, field unification and other issues are dealt with through data standardization methods, and an abnormal call identification model is proposed to achieve real-time, fast and high-precision detection on the API gateway side. By improving the loss function of Xgboost, the problem of multivariate screening under the same data structure is solved, and the feature group with a greater degree of influence on the model can be selected more accurately. In the process of constructing a decision tree, the improved two-layer random forest model reduces the possibility of overfitting problems by constraining the growth structure of the decision tree, and can more accurately perform target screening and feature extraction.

[0125] The improved two-layer random forest algorithm is suitable for anomaly identification of API gateway interface calls. The purpose of proposing a high-fitting two-layer random forest algorithm is to take into account the KPI performance data and log information data generated by the call at the same time. By building a two-layer classifier and performing sub-forest screening of the random forest model, fast and high-accuracy detection of call anomalies can be achieved. The improved random forest algorithm is also an ensemble learning algorithm of the decision tree. It has high interpretability and outputs variable weights through the process of building a decision tree. The improved random forest algorithm makes the model effect more accurate, further eliminates multicollinearity between variables, and improves the efficiency of model operation.

[0126] Compared with the traditional discovery method of abnormal identification, the main advantage of the present invention is that it breaks the traditional way of identifying abnormal calls by setting rules. Due to the problems of unreasonable rule setting and too long rule update cycle in this artificial setting, the accuracy of abnormal call identification is low and the actual use value of the function is not high. The present invention further standardizes the recognition results through a two-layer random forest model, so that the results are interpretable and highly accurate. In response to the demand for real-time detection of abnormal calls on the API gateway side, modeling for performance resource data is proposed. API calls are detected by the performance resource detection model and the selected improved random forest algorithm, and heterogeneous calls are captured in combination with performance data indicators to clarify the deep meaning of the fluctuations in performance resource data. The detection of abnormal calls is completed efficiently and accurately.

[0127] Further, see Figure 7 As shown, the present invention also provides a system for monitoring abnormal calls of a big data platform API gateway, which is applied to a method for monitoring abnormal calls of a big data platform API gateway described in the above embodiment to realize monitoring abnormal calls of a big data platform API gateway and identifying abnormal calls. The system includes:

[0128] The data acquisition module is used to obtain the log data and performance indicator data of the API gateway call of the big data platform;

[0129] The feature screening and data set construction module is used to screen the feature fields of log data through the Xgboost algorithm, and construct a new input data set and feature set in combination with performance indicator data;

[0130] The anomaly recognition model building module is used to train the improved two-layer random forest model using the data set and feature set to build an anomaly recognition model;

[0131] The abnormal call identification module is used to deploy the established abnormal call identification model on the API gateway side, capture call logs, IP and related performance data online, input the abnormal call identification model, and complete abnormal call identification on the gateway side.

[0132] The embodiment of the present invention provides a system for monitoring abnormal calls of the API gateway of a big data platform. Its implementation principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0133] In addition, refer to Figure 8As shown, an embodiment of the present invention also provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, and the processor executes the computer program to implement a method for monitoring abnormal calls to a big data platform API gateway in the above method embodiment.

[0134] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0135] The memory 11 may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination thereof.

[0136] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, electronic devices or computer program products, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0137] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0138] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0139] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring abnormal calls of API gateways on a big data platform, characterized in that: The method comprises the following steps: S1. Obtain the log data and performance indicator data of the API gateway call of the big data platform; S2. Use the Xgboost algorithm to perform feature screening on the feature fields of the log data, and combine the performance indicator data to construct a new input data set and feature set; S3. Use the data set and feature set to train the improved two-layer random forest model and establish an anomaly recognition model; S4. Deploy the established exception recognition model on the API gateway side, capture call logs, IP and related performance data online, input the exception recognition model, and complete abnormal call recognition on the gateway side.

2. According to claim 1, a method for monitoring abnormal calls of a big data platform API gateway is characterized in that: In S1, the obtained log data includes: access time, http access path, returned status code information and method name; each log sample generates specific fields with the user's calling address and calling time as a record, and at the same time, the second-level performance indicator data when each log occurs is associated with the log feature through time features; the performance indicator data includes: CPU utilization, second-level concurrency and server response time.

3. According to claim 1, a method for monitoring abnormal calls of a big data platform API gateway is characterized in that: In S2, feature screening is performed based on the Xgboost algorithm with structural risk terms added. The objective function of the Xgboost algorithm with structural risk terms added is: Among them, g(x) represents the objective function, y i represents the true value, represents the predicted value, i represents the i-th sample, n represents the number of samples, L represents the loss function, fk represents the structural risk term, K represents the number of trees, T represents the number of leaf nodes, w represents the weight of the node, and a and b are hyperparameters.

4. According to claim 3, a method for monitoring abnormal calls of a big data platform API gateway is characterized in that: In S3, the process of establishing anomaly recognition model includes: 1) Extract K training data sets from the data set with replacement. The number of samples in each training subset is N. These samples are called bootstrap samples to train the decision trees respectively. 2) Use cart classification and regression tree to construct a decision tree. At the node of the tree, randomly select m features from M input features as the split feature set of the current node of the decision tree, select the optimal split feature and split point, and assign the training data set to the child node; 3) Train K bootstrap sample sets in sequence and obtain all decision trees according to the set parameters; 4) Prune the forest, construct a two-layer random forest model, select the optimal subtree, select the top N abnormal categories of the probability of the cart decision tree, and introduce a counter i=1 when the predicted value output by the model is equal to the true value, that is, it returns correct. When the true label is not equal to the predicted label, the counter i=0; by constructing the loss function, find out several decision trees whose loss function is less than the target value to form a sub-forest.

5. A method for monitoring abnormal calls of a big data platform API gateway according to claim 4, characterized in that: The optimal splitting features and splitting points are selected using the Gini coefficient minimization criterion.

6. A method for monitoring abnormal calls of a big data platform API gateway according to claim 1, characterized in that: The performance of the anomaly recognition model is verified through precision, recall, PR curve, F1 score, ROC curve and AUC value.

7. A system for monitoring abnormal calls of API gateways on a big data platform, characterized in that: When applied, a method for monitoring abnormal calls of a big data platform API gateway according to any one of claims 1 to 6 is executed to realize abnormal call identification, and the system includes: The data acquisition module is used to obtain the log data and performance indicator data of the API gateway call of the big data platform; The feature screening and data set construction module is used to screen the feature fields of log data through the Xgboost algorithm, and construct a new input data set and feature set in combination with performance indicator data; The anomaly recognition model building module is used to train the improved two-layer random forest model using the data set and feature set to build an anomaly recognition model; The abnormal call identification module is used to deploy the established abnormal call identification model on the API gateway side, capture call logs, IP and related performance data online, input the abnormal call identification model, and complete abnormal call identification on the gateway side.

8. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a method for monitoring abnormal calls of a big data platform API gateway as described in any one of claims 1-6.

Images