An API endpoint management method and system based on unsupervised learning

Abstract

This application relates to the field of network security technology, and more specifically to an unsupervised learning-based API endpoint management method and system. The method can extract Host and Path information from passive traffic to form a URL-based API endpoint to be managed, and can also extract relevant URL features to perform two-level grouping to achieve unsupervised management of incremental APIs.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and specifically to an API endpoint management method and system based on unsupervised learning. Background Art

[0002] In today's era of rapid internet development, communication between various applications and systems is becoming increasingly frequent and intensive. APIs (Application Programming Interfaces) play a crucial role as a bridge between different applications. They integrate the functionality of different applications, allowing developers to easily access functionality from within their own applications. For example, integrating the API of a third-party payment platform within a website allows for easy payment functionality. This enables countless systems, platforms, and applications to work together more efficiently, significantly improving the efficiency of data processing, transmission, and analysis across various businesses.

[0003] With the widespread use of APIs, their security issues are becoming increasingly prominent. API endpoints, as specific functions within an API service, are typically represented by HTTP request URL paths. An HTTP request is a request message sent by a browser to an HTTP server. Each HTTP request consists of a request line, header fields, and an optional entity body. In real-world applications, the number of API endpoints is numerous and will continue to increase as business grows. Accurately identifying and managing API endpoints is critical to ensuring system security.

[0004] However, traditional API endpoint identification methods have many problems:

[0005] First, labor costs are high. Traditional methods rely on manual configuration and management, often relying on active scanning to discover endpoints when source code is unavailable. This not only consumes significant manpower, material resources, and time, but as the number of API endpoints grows, manual processing becomes difficult to ensure accuracy and real-time performance.

[0006] Second, it cannot automatically adapt to changes. Existing technologies require manual updates to configuration and management policies to address dynamic changes in API endpoints, which can easily lead to false positives and false negatives, impacting identification effectiveness and security.

[0007] Third, recognition accuracy is insufficient. Existing technologies are susceptible to factors such as data noise and insufficient sampling, resulting in poor recognition accuracy, making it difficult for security teams to promptly detect and locate potential security risks.

[0008] Fourth, endpoint management is inefficient. Traditional methods, when processing large amounts of data, take a long time to compare, classify, and merge API endpoints. They also fail to achieve real-time, sustainable, or incremental recognition. Importing and merging a large number of URLs repeatedly each time is extremely inefficient.

[0009] Fifth, active scanning findings are incomplete. Without a high-privileged administrative account, the API endpoints discovered through active scanning are often incomplete. Summary of the Invention

[0010] Given these issues with existing technologies, researching an API endpoint identification method and system based on unsupervised learning is of great practical significance. Unsupervised learning, as a machine learning method, can automatically discover hidden patterns and structures in data without prior label information, providing new insights and directions for solving the API endpoint identification problem.

[0011] The present application discloses an API endpoint management method based on unsupervised learning. The method may include: obtaining traffic data; extracting Host and Path information from the traffic data and forming a to-be-managed API endpoint in the form of a URL; extracting URL features from the to-be-managed API endpoint; the URL features include protocol features, host features, path level features, and path abstract features of each path level; dividing the to-be-managed API endpoint into target groups based on the URL features; the URL features of the API endpoints in the target group are the same as the URL features of the to-be-managed API endpoint; performing secondary grouping of the API endpoints in the target group based on the URLs of the API endpoints in the target group to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are no more than a first preset number.

[0012] In some embodiments, the method further includes: for each final group, generating an API endpoint model corresponding to the final group based on the URL of the API endpoint in the final group; extracting URL features from the API endpoint to be managed includes: extracting URL features from the API endpoint to be managed when the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint.

[0013] In some embodiments, the method further includes: extracting security elements associated with the API endpoint to be managed from the traffic data; updating a security baseline white model based on the security elements; and using the security baseline white model to perform security detection on the traffic.

[0014] In some embodiments, the method further includes: obtaining an API interface document; comparing the API endpoints in each of the final groups with the endpoints in the API interface document, and determining the API endpoints that do not exist in the API interface document as abnormal API endpoints.

[0015] In some embodiments, the method further includes: after generating the API endpoint model corresponding to the final group, replacing the original API endpoint model of the final group, and saving the original API endpoint model to perform model fallback in the event that the newly generated API model does not match accurately.

[0016] In some embodiments, the extracting of URL features from the API endpoint to be managed includes: using a preset identifier to split the URL of the API endpoint to be managed from left to right to obtain at least one split feature; determining the first split feature obtained as a protocol feature; determining the second split feature obtained as a host feature; determining the remaining number of split features as path level features; and for each of the remaining split features, determining the abstracted regular expression as the path abstract feature of the corresponding path level.

[0017] In some embodiments, the dividing the API endpoints to be managed into target groups based on the URL characteristics includes: generating summary information of the API endpoints to be managed based on the URL characteristics; dividing the API endpoints to be managed into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0018] In some embodiments, the API endpoints in the target group are secondary grouped according to the path abstract features of each path level of the API endpoints in the target group to obtain at least one final group, including: for each API endpoint in the target group, its URL is compared with the URLs of other API endpoints path level by path level, and a first API endpoint whose values ​​at each path level are exactly the same or a second API endpoint whose value is different only at one path level is determined, and the API endpoints, the first API endpoint, and the second API endpoint are divided into a final group.

[0019] In some embodiments, generating the API endpoint model corresponding to the final group based on the URL and URL features of the API endpoint in the final group includes: replacing different values ​​in the URL of the API endpoint in the final group with corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

[0020] This application also proposes an API endpoint management system based on unsupervised learning. The system includes: a traffic collection unit for acquiring traffic data; an API endpoint parsing unit for extracting Host and Path information from the traffic data and forming an API endpoint to be managed in the form of a URL; an API endpoint merging unit for extracting URL features from the API endpoint to be managed; the URL features including protocol features, host features, path level features, and path abstract features of each path level; and, based on the URL features, dividing the API endpoints to be managed into target groups; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoints to be managed; and, based on the URLs of the API endpoints in the target group, secondary grouping the API endpoints in the target group to obtain at least one final group; the URLs corresponding to the API endpoints in the final group differ in path level values ​​by no more than a first preset number; and, for each final group, generating an API endpoint model corresponding to the final group based on the URLs of the API endpoints in the final group; an API endpoint storage unit for storing the URLs of the API endpoints; an API endpoint white model modeling unit for generating a security baseline white model; and a detection unit for performing security detection on the traffic based on the security baseline white model.

[0021] In the scheme recorded in the above-mentioned embodiment, Host and Path information can be extracted from the traffic data to form an API endpoint to be managed in the form of a URL, and then URL features are extracted from the API endpoint to be managed and, based on the URL features, the API endpoint to be managed is divided into target groups with the same URL features, and based on the URLs of the API endpoints in the target group, the API endpoints in the target group are secondary grouped to obtain at least one final group; wherein, the values ​​of the path levels corresponding to the URLs of the API endpoints in the final group are different by no more than a first preset number.

[0022] Therefore, firstly, the identification and management of API endpoints through passive traffic and unsupervised learning methods reduces the cost of manual import and configuration, while improving the efficiency of API endpoint identification and management.

[0023] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0024] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, API endpoints can be managed in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0025] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group to which it belongs can be quickly found. Further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large number or all API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance consumption and improves management efficiency.

[0026] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The following is a brief introduction to the drawings required for describing the embodiments or related technologies.

[0028] Figure 1 A flowchart of an API endpoint management method based on unsupervised learning is shown in an embodiment of the present application.

[0029] Figure 2 A flowchart of a method for extracting URL features illustrated in this application.

[0030] Figure 3 This is a flow chart of a method for establishing a security baseline white model shown in this application.

[0031] Figure 4 This is a flow chart of the method for identifying abnormal API endpoints illustrated in this application.

[0032] Figure 5 This is a flowchart of the method for grouping by abstracts illustrated in this application.

[0033] Figure 6 This is a schematic diagram of the structure of the API endpoint management system illustrated in this application.

[0034] Figure 7 This is a flowchart of the API endpoint management method illustrated in this application. DETAILED DESCRIPTION

[0035] The following exemplary embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0036] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used herein, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0037] Given the challenges of existing technologies, researching an API endpoint identification method and system based on unsupervised learning is of great practical significance. Unsupervised learning, as a machine learning method, can automatically discover hidden patterns and structures in data without prior label information, providing new insights and directions for solving the API endpoint identification problem.

[0038] This application proposes an API endpoint management method based on unsupervised learning. The method can extract Host and Path information from passive traffic to form the API endpoint to be managed in the form of URLs, and can also achieve unsupervised management of incremental APIs by extracting relevant URL features and performing two-level grouping.

[0039] The following is an example description with reference to the accompanying drawings. Figure 1 , Figure 1 A flowchart of an API endpoint management method based on unsupervised learning is shown in an embodiment of the present application.

[0040] Figure 1 The illustrated unsupervised learning-based API endpoint management method can be applied to electronic devices. The electronic devices can execute the method by implementing software logic corresponding to the unsupervised learning-based API endpoint management method. The electronic devices can be laptops, computers, servers, mobile phones, PDAs, and the like. The type of electronic device is not particularly limited in this application. The electronic devices can also be client devices or server devices.

[0041] like Figure 1 As shown, the method may include S102 to S110. Unless otherwise specified, the present application does not particularly limit the execution order of these steps.

[0042] S102, obtaining traffic data.

[0043] Traffic can include request traffic, response traffic, and HTTP / HTTPS traffic. Request traffic is generated when a client sends a request to a server, while response traffic is generated when a server responds to a request. HTTP / HTTPS traffic is traffic for commonly used network communication protocols.

[0044] When acquiring traffic data for HTTP traffic, you can use a packet capture tool like Wireshark or a probe to capture packets directly on the network interface and extract traffic data. For example, within an enterprise network, you can install Wireshark on the gateway connecting the internal and external networks and set filtering conditions to capture HTTP traffic.

[0045] For HTTPS traffic, due to its encryption characteristics, you need to first configure an SSL decryption device on the client or server side. After legal decryption, use a packet capture tool to extract data.

[0046] Passive traffic, mirrored traffic, and non-active scanning traffic are primarily captured by configuring port mirroring on network devices such as switches. For example, in a large network, mirroring traffic from switch ports connected to important servers to an electronic device can capture this traffic data for subsequent AIP endpoint identification and discovery.

[0047] S104: Extract Host and Path information from the traffic data and compose an API endpoint to be managed in the form of a URL.

[0048] In an HTTP request, the host field identifies the domain name or IP address of the target server. It's a crucial component of the URL and determines which server the request will be sent to. For example, in the request "https: / / www.example.com / api / data," "www.example.com" is the host field. This field allows the client to accurately locate the server, ensuring the data transfer is targeted.

[0049] Path information refers to the path to a specific resource on the target server. Continuing with the request above, " / api / data" is the path information, which specifies the specific resource location to be accessed on the server "www.example.com." The path can be used to locate a specific file, directory, or API interface on the server.

[0050] Extracting Host and Path information from traffic data and composing them into a URL for the API endpoint to be managed involves the following steps: First, parse the HTTP request message from the captured traffic data. In the message, the Host information is typically located in the request header field, beginning with "Host:" followed by the server's domain name or IP address. For example, in the following request header:

[0051] GET / api / data HTTP / 1.1;

[0052] Host: www.example.com;

[0053] User-Agent: Mozilla / 5.0;

[0054] You can easily extract "www.example.com" as the Host information.

[0055] The Path information usually follows the HTTP request method (such as GET, POST, etc.). In the above example, the " / api / data" following "GET" is the Path information.

[0056] After extracting the Host and Path information, construct a URL using the format "protocol: / / host information:port number / path information." For HTTP, the default port number is 80; for HTTPS, the default port number is 443. When using the default port number, the port number can be omitted. For example, the URL "https: / / www.example.com / api / data" extracted above is formed into "https: / / www.example.com / api / data." This URL can be used as the API endpoint to be managed, for subsequent API endpoint identification, management, and security monitoring.

[0057] S106: Extracting URL features from the API endpoint to be managed.

[0058] URL features are features extracted based on the URL of the API endpoint to be managed. They can indicate the specific characteristics of the endpoint and facilitate subsequent group management. URL features include protocol features, host features, path level features, and path abstraction features for each path level.

[0059] The present application can use various methods to extract the aforementioned ULR features, for example, displaying the URL and manually splitting it, but this method is less efficient.

[0060] In some embodiments, automatic feature extraction can be used. Figure 2 , Figure 2 This is a flow chart of a method for extracting URL features as shown in this application. This method is an implementation of S106. Figure 2 As shown, the method may include S202-S210.

[0061] S202, using a preset identifier to split the URL of the API endpoint to be managed from left to right to obtain at least one splitting feature;

[0062] S204, determining the first obtained split feature as a protocol feature;

[0063] S206, determining the second obtained split feature as the host feature;

[0064] S208, determining the remaining number of split features as path-level features;

[0065] S210 : For each of the remaining split features, determine the abstracted regular expression as a path abstraction feature of the corresponding path level.

[0066] Taking the URL of the API endpoint to be managed, "http: / / www.example.com / api / v1 / login", as an example, the steps to extract its URL features are as follows:

[0067] S202: Split the URL using predefined identifiers. Predefined identifiers are typically ": / / " and " / ." First, split the URL from left to right based on ": / / " into "http" and "www.example.com / api / v1 / login." Then, further split "www.example.com / api / v1 / login" based on " / " to obtain the split features "www.example.com," "api," "v1," and "login."

[0068] S204, determining the protocol feature: the first split feature obtained is the protocol feature, so the protocol feature of this URL is "http", which indicates that the data transmission adopts the Hypertext Transfer Protocol.

[0069] S206, determining the host feature: The second split feature obtained is the host feature, ie "www.example.com", which represents the address of the server providing the service. The corresponding server can be located through the host name to obtain the relevant API service.

[0070] S208, determine the path level feature: After determining the protocol and host features, the remaining split features are "api", "v1", and "login", a total of 3, so the path level feature is 3, which reflects the complexity of the API endpoint in the path structure and the hierarchical division of functional organization.

[0071] S210, determining the path abstract features:

[0072] For "api", it can be abstracted into a regular expression "[az]", with the number of characters being 3, indicating that this level is composed of 3 lowercase letters.

[0073] For "v1", it can be abstracted as "[az] [0-9]", and the number of characters is 2, indicating that this level consists of 1 lowercase letter and 1 number.

[0074] For "login", it can be abstracted as "[az]", and the number of characters is 5, indicating that this level is composed of 5 lowercase letters.

[0075] Thus, through S202-S210, automatic URL feature extraction can be achieved, which facilitates subsequent endpoint management. In some methods, feature extraction algorithms such as neural networks can also be used for path abstract features.

[0076] S108: Divide the API endpoints to be managed into target groups according to the URL features.

[0077] This step is the first grouping of API endpoints. The URL characteristics of the API endpoints in the target group are the same as the URL characteristics of the API endpoints to be managed.

[0078] In this step, a URL feature matching method may be used. For example, URL features may be combined into a vector and the target group may be found through Mahalanobis distance, cosine distance, and the like.

[0079] It is understood that in this step, if a target group with the same characteristics as the URL can be matched, the endpoint to be managed can be placed in the target group. If no target group is matched, a group can be created for the API endpoint to be managed.

[0080] S110 , performing secondary grouping on the API endpoints in the target group according to the URLs of the API endpoints in the target group to obtain at least one final group.

[0081] The URLs corresponding to the API endpoints in the final group have no more than a first preset number of path level differences. The first preset number is set based on demand. For example, it can be 1 or 2. Assuming the first preset number is 1, after the second grouping, every two URLs in each final group are identical, or only differ in one level of path abstraction features.

[0082] Taking the first preset number as 1 as an example, in S110, for each API endpoint in the target group, its URL can be compared with the URLs of other API endpoints path by path level to determine the first API endpoint with exactly the same value at each path level or the second API endpoint with only one path level value different.

[0083] The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

[0084] For example, suppose the target group contains the following API endpoint URLs:

[0085] https: / / www.example.com / api / v1 / user / 100.

[0086] https: / / www.example.com / api / v1 / user / 200.

[0087] https: / / www.example.com / api / v1 / user / 300.

[0088] https: / / www.example.com / api / v1 / profile / 500.

[0089] https: / / www.example.com / api / v1 / profile / 600.

[0090] https: / / www.example.com / api / v1 / profile / 700.

[0091] We set the first preset number to 1, which means that after the second grouping, every two URLs in each final group are either completely identical or have only one level of path abstract features that are different.

[0092] Specifically, for each API endpoint in the target group, its URL is compared with the URLs of other API endpoints path by path level.

[0093] Take "https: / / www.example.com / api / v1 / user / 100" as an example.

[0094] Compared with "https: / / www.example.com / api / v1 / user / 200", their protocol features are both "https", their host features are both "www.example.com", and their path level features "api", "v1", and "user" are all the same. Only the last level number "100" and "200" are different. In other words, only one path level value is different, which meets the conditions.

[0095] Similarly, "https: / / www.example.com / api / v1 / user / 100"

[0096] This URL is different from "https: / / www.example.com / api / v1 / user / 300" only in the last digit, so it meets the requirements.

[0097] "https: / / www.example.com / api / v1 / user / 100"

[0098] For example, in the example "https: / / www.example.com / api / v1 / profile / 500", the first three layers "https", "www.example.com", "api", and "v1" are the same, but the fourth layer "user" and "profile" are different, and the last layer "100" and "500" are also different. The values ​​of two path layers are different, so the condition is not met.

[0099] So, you can change "https: / / www.example.com / api / v1 / user / 100",

[0100] "https: / / www.example.com / api / v1 / user / 200" and "https: / / www.example.com / api / v1 / user / 300" are grouped into one final group.

[0101] Change "https: / / www.example.com / api / v1 / profile / 500",

[0102] "https: / / www.example.com / api / v1 / profile / 600",

[0103] "https: / / www.example.com / api / v1 / profile / 700" is classified into one final group.

[0104] By continuously comparing and dividing all API endpoints within the target group in the above manner, at least one final group can be obtained. The URLs of the API endpoints within each final group satisfy the condition that the values ​​of the path levels are different by no more than a first preset number, thereby realizing group management of the API endpoints.

[0105] Through the scheme recorded in S102-S110, Host and Path information can be extracted from traffic data and composed into API endpoints to be managed in the form of URLs. Then, URL features are extracted from the API endpoints to be managed and, based on the URL features, the API endpoints to be managed are divided into target groups with the same URL features. Furthermore, based on the URLs of the API endpoints in the target groups, the API endpoints in the target groups are secondary grouped to obtain at least one final group. The URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are no more than a first preset number.

[0106] Therefore, firstly, the identification and management of API endpoints through passive traffic and unsupervised learning methods reduces the cost of manual import and configuration, while improving the efficiency of API endpoint identification and management.

[0107] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0108] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, API endpoints can be managed in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0109] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group to which it belongs can be quickly found. Further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large number or all API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance consumption and improves management efficiency.

[0110] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive.

[0111] In some embodiments, the API endpoints under the final group can be modeled. If a newly identified API endpoint that matches the model is found, it can be released. If a newly identified API endpoint that does not match the model is found, it can be analyzed in depth. If it is analyzed to be a risky access, it can be intercepted and an alarm can be issued immediately. If it is found to be a safe access, it can be released and incremental management can be performed using solutions such as S106-S110.

[0112] In some embodiments, after S110, the method further includes:

[0113] For each final group, an API endpoint model corresponding to the final group is generated according to the URL of the API endpoint in the final group.

[0114] In this step, different values ​​in the URLs of the API endpoints in the final group may be replaced with corresponding regularized expressions to obtain an API endpoint model corresponding to the final group.

[0115] Take the final grouping divided above as an example. The URLs contained in the first final group are:

[0116] https: / / www.example.com / api / v1 / user / 100

[0117] https: / / www.example.com / api / v1 / user / 200

[0118] https: / / www.example.com / api / v1 / user / 300

[0119] In these URLs, the protocol "https", the host "www.example.com", and the first three layers of the path hierarchy "api", "v1", and "user" are all fixed, and only the numbers in the last layer are different. These different numbers represent different user identifiers. In order to abstract this set of URLs into a common API endpoint model, the changing part (that is, the numbers in the last layer) can be replaced with the corresponding regularized expression. Since this part is a number, the regular expression "[0-9]" can be used, which means matching one or more numbers. Therefore, the API endpoint model corresponding to this final group is: https: / / www.example.com / api / v1 / user / [0-9]. Through this model, the common characteristics and changing patterns of this group of API endpoints can be accurately represented, where "[0-9]" covers all possible numerical values ​​that may appear at this path level.

[0120] The second final group contains the URL:

[0121] https: / / www.example.com / api / v1 / profile / 500

[0122] https: / / www.example.com / api / v1 / profile / 600

[0123] https: / / www.example.com / api / v1 / profile / 700

[0124] Similarly, the protocol, host, and path levels remain unchanged, while the last level's numbers vary, representing different profile identifiers. Replacing these varying values ​​with the regular expression "[0-9]" yields the API endpoint model for this final group: https: / / www.example.com / api / v1 / profile / [0-9]. This model captures the commonalities of this group of API endpoints.

[0125] By replacing different values ​​in the URLs of the API endpoints in the final group with corresponding regularized expressions to generate an API endpoint model, the characteristics of the API endpoints in the final group can be described in a concise and universal way, which plays an important role in API management, analysis, and security monitoring.

[0126] In S106, extracting URL features from the API endpoint to be managed includes:

[0127] In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

[0128] For example, taking the aforementioned API endpoint-related content as an example, assume that there are already generated API endpoint models, such as "https: / / www.example.com / api / v1 / user / [0-9]" and "https: / / www.example.com / api / v1 / profile / [0-9]".

[0129] When a new API endpoint to be managed appears, the first thing to do is determine whether it matches any existing API endpoint models. For example, if the new API endpoint to be managed is "https: / / www.example.com / api / v2 / product / 123," we compare it to the existing models. The protocol characteristics are both "https" and the host characteristics are both "www.example.com." However, in the path hierarchy characteristics, the version number "v2" after "api" differs from "v1" in the existing model, and "product" differs from both "user" and "profile" in the existing model. Therefore, it does not match any existing API endpoint models.

[0130] Furthermore, relevant technologies can be used to determine whether the API endpoint to be managed is an abnormal endpoint. Abnormal endpoints may include those with format errors, those that do not comply with basic network communication standards, or malicious requests from unknown sources. If the "https: / / www.example.com / api / v2 / product / 123" endpoint is found to be an abnormal endpoint, the relevant access can be intercepted and an alert can be issued. If it is not an abnormal endpoint, URL features can be extracted from the API endpoint to facilitate subsequent endpoint management.

[0131] Therefore, the API endpoints in the final group can be modeled to obtain a model that can characterize the features and implement risk detection.

[0132] In some embodiments, after the API endpoint model corresponding to the final group is generated, the original API endpoint model of the final group is replaced and the original API endpoint model is saved to perform model fallback in the event that the newly generated API model does not match accurately.

[0133] In the API endpoint management process, after the API endpoint model corresponding to a specific final group is generated, model replacement and retention operations need to be performed.

[0134] Specifically, the newly generated API endpoint model can replace the original API endpoint model for the final group. This allows for more accurate and efficient management and analysis of API endpoints. The new model is typically based on more comprehensive data and in-depth analysis, more accurately reflecting the actual state of the API endpoints.

[0135] In real-world scenarios, newly generated API models may experience mismatches. For example, in certain scenarios, the new model may not accurately identify API endpoints or may misinterpret traffic data. If this happens, a model rollback can be performed, re-applying the original, retained API endpoint model. This rollback ensures continued normal management and analysis of API endpoints, mitigating potential misjudgments and security risks caused by the new model's inaccuracies.

[0136] In some embodiments, a security baseline white model can be established based on the security elements of the API endpoint to perform security testing. Figure 3 , Figure 3 This is a flow chart of a method for establishing a security baseline white model shown in this application. Figure 3 As shown, it may include S302-S304.

[0137] S302: Extract security elements associated with the API endpoint to be managed from the traffic data.

[0138] S304: Update the security baseline white model based on the security elements; the security baseline white model is used to perform security detection on traffic.

[0139] The security elements can be extracted from HTTP messages such as request URLs, request messages, and response messages, including information such as the API transmission data format, API transmission data size, API transmission latency, number of API parameter names, number of API paths, and API parameter values. These elements can be standardized and combined with pre-set rules to form a security baseline white model. The security baseline white model can be an artificial intelligence model, including models built using linear regression, neural networks, deep learning, or large models.

[0140] The security baseline white model can include various metric ranges for normal traffic. For example, under normal circumstances, the size of API data transmission may be between 1KB and 50KB, and the API transmission latency may be between 100ms and 300ms. When relevant metrics in traffic data exceed these ranges, it may indicate risky traffic. For example, if the data size suddenly soars to 100KB or the transmission latency increases to 1000ms, these anomalies may indicate a network attack, such as malicious large-scale data injection or a denial of service attack causing delayed response.

[0141] Assume we have an API endpoint to be managed, "https: / / www.example.com / api / v1 / product / query", and the traffic data associated with it:

[0142] API transmission data format: If the request message is {"category":"electronics","priceRange":[100, 500]}, the data format is determined to be JSON.

[0143] API transmission data size: After calculation, the request message size is 40 bytes.

[0144] API transmission latency: It takes 250 milliseconds from sending a request to receiving a response.

[0145] Number of API parameter names: There are two parameter names in the message: "category" and "priceRange".

[0146] Number of API paths: For the request URL, the number of path levels is 4 ("api", "v1", "product", "query").

[0147] API parameter values: For example, the value of "category" is "electronics" and the value of "priceRange" is "[100, 500]".

[0148] After obtaining the relevant security factors, the security baseline white model can be updated using methods such as linear regression. For example, using the size of API transmission data as an example, a linear regression algorithm is used to build a model using the data size and corresponding normal / non-normal labels from a large amount of historical traffic data as a training set. Assuming the current extracted API transmission data size is 40 bytes, incorporating this data into the model for calculation yields a range of 1KB to 60KB (hypothetical values; actual calculations will vary based on the algorithm and data). By continuously incorporating newly extracted security factors into the linear regression model for calculation and update, the security baseline white model can accurately reflect the characteristics of normal traffic, effectively conducting security checks on subsequent traffic and promptly identifying potential security risks.

[0149] In some embodiments, abnormal endpoints can also be identified among the managed API endpoints, including zombie API endpoints and shadow API endpoints. Figure 4 , Figure 4 This is a flow chart of the method for identifying abnormal API endpoints illustrated in this application.

[0150] S402, obtain API interface documentation.

[0151] S404: Compare the API endpoints in each final group with the endpoints in the API interface document, and determine the API endpoints that do not exist in the API interface document as abnormal API endpoints.

[0152] The interface documentation refers to recorded known API endpoints, which can be maintained manually or by the system.

[0153] In this embodiment, the URL of the API endpoint within each final group can be compared with the endpoint URL in the retrieved API interface documentation. For example, if the URL is https: / / www.example.com / api / v1 / custom / operation and no matching endpoint record is found in the API interface documentation, this endpoint is considered abnormal and an alert is issued. Normal endpoints can include zombie API endpoints and shadow API endpoints.

[0154] Identifying these abnormal API endpoints can effectively prevent potential security risks.

[0155] In some embodiments, in S108 , a method of calculating a summary may be used to perform the first step of grouping, thereby reducing the amount of calculation, improving grouping efficiency, and further improving the efficiency of API endpoint management.

[0156] See Figure 5 , Figure 5This is a flowchart of the method for grouping by abstracts illustrated in this application.

[0157] S502: Generate summary information of the API endpoint to be managed according to the URL feature.

[0158] S504 , dividing the API endpoints to be managed into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0159] The method for calculating the summary information can refer to the summary algorithm in the relevant calculation, such as MD5, SHA-1, etc.

[0160] Take http: / / www.example.com / api / v1 / login as an example, through Figure 2 The URL features extracted by this illustrated method include: protocol feature "http", host feature "www.example.com", and path abstract features "[az]" (3 lowercase letters), "[az][0-9]" (1 lowercase letter and 1 number), and "[az]" (5 lowercase letters). MD5 can be used to calculate the digest information. This digest information can be used to complete the first step of grouping.

[0161] This application also proposes an API endpoint management system based on unsupervised learning. Figure 6 , Figure 6 This is a schematic diagram of the structure of the API endpoint management system illustrated in this application. Figure 6 As shown, the system 600 may include:

[0162] Traffic collection unit 610, used to obtain traffic data;

[0163] An API endpoint parsing unit 620 is configured to extract Host and Path information from the traffic data and compose an API endpoint to be managed in the form of a URL;

[0164] The API endpoint merging unit 630 is configured to extract URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstraction features of each path level;

[0165] and, dividing the API endpoints to be managed into target groups according to the URL characteristics; the API endpoints in the target groups have the same URL characteristics as the URL characteristics of the API endpoints to be managed;

[0166] and, performing secondary grouping of the API endpoints in the target group according to the URLs of the API endpoints in the target group to obtain at least one final group; wherein the URLs corresponding to the API endpoints in the final group differ in path level values ​​by no more than a first preset number;

[0167] And, for each final group, generating an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group;

[0168] API endpoint storage unit 640, used to store the URL of the API endpoint;

[0169] API endpoint white model modeling unit 650, for generating a security baseline white model;

[0170] The detection unit 660 is configured to perform security detection on traffic based on the security baseline white model.

[0171] For the relevant description of the steps executed by each unit of the system, reference may be made to any of the previous method embodiments.

[0172] In some embodiments, the API endpoint merging unit 630 further:

[0173] In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

[0174] In some embodiments, the API endpoint storage unit 640 further extracts security elements associated with the API endpoint to be managed from the traffic data;

[0175] The API endpoint white model modeling unit 650 further updates the security baseline white model based on the security elements; the security baseline white model is used to perform security detection on traffic.

[0176] In some embodiments, the detection unit 660 further obtains an API interface document;

[0177] The API endpoints in each of the final groups are compared with the endpoints in the API interface document, and the API endpoints that do not exist in the API interface document are determined as abnormal API endpoints.

[0178] In some embodiments, the API endpoint storage unit 640 further replaces the original API endpoint model of the final group after generating the API endpoint model corresponding to the final group, and saves the original API endpoint model to perform model fallback in the event that the newly generated API model does not match accurately.

[0179] In some embodiments, the API endpoint merging unit 630 further:

[0180] Splitting the URL of the API endpoint to be managed from left to right using a preset identifier to obtain at least one splitting feature;

[0181] The first obtained split feature is determined as the agreement feature;

[0182] The second obtained split feature is determined as the host feature;

[0183] The number of remaining split features is determined as path-level features;

[0184] For each of the remaining split features, the abstracted regular expression is determined as the path abstraction feature of the corresponding path level.

[0185] In some embodiments, the API endpoint merging unit 630 further:

[0186] Generate summary information of the API endpoint to be managed based on the URL characteristics;

[0187] The API endpoints to be managed are divided into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0188] In some embodiments, the API endpoint merging unit 630 further:

[0189] For each API endpoint in the target group, compare its URL with the URLs of other API endpoints path by path level to determine a first API endpoint with the same value at each path level or a second API endpoint with only one path level value different.

[0190] The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

[0191] In some embodiments, the API endpoint merging unit 630 further:

[0192] The different values ​​in the URLs of the API endpoints in the final group are replaced with corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

[0193] See Figure 7 , Figure 7 This is a flowchart of the API endpoint management method illustrated in this application. Figure 7 The schematic method can be based on Figure 6 Schematic API endpoint management system implementation. Figure 7 As shown, the method may include S701-S717.

[0194] S701, start.

[0195] S702, collect HTTP traffic through the traffic collection unit 610, and some HTTPS traffic that supports decryption can also be collected, and then enter S703.

[0196] S703, through the API endpoint parsing unit 620, extract the request Host and path from the traffic and compose the URL complete path to obtain the AIP endpoint to be managed, and enter S704.

[0197] S704 - S711 and S713 - S715 are executed by the API endpoint merging unit 630 .

[0198] S704: Determine whether it is an API request based on the relevant information in the traffic. If so, proceed to S705; if not, proceed to S717 and the process ends.

[0199] S705: Split the URL path of the AIP endpoint to be managed into units of : / / or / to extract URL features. The URL features include protocol features, host features, path level features, and path abstract features for each path level. Then proceed to S706.

[0200] S706 , performing digest calculation based on the combined URL features obtained in the previous step, obtaining a unique digest result of the AIP endpoint to be managed, and proceeding to S707 .

[0201] S707, determine whether it is incremental training, if yes, execute S708; if not, execute S713.

[0202] S708, determining whether the summary generated by the AIP endpoint to be managed already has a historical group. If so, proceed to S709; if not, proceed to S713.

[0203] S709: Store the AIP endpoint to be managed into a target group with the same summary, and proceed to S710.

[0204] S710 , performing secondary grouping on the AIP endpoints to be managed in the target group, finding AIP endpoints with only unique different values ​​or all the same values, and grouping them into one group to obtain at least one final group, so as to enter S711 for further processing.

[0205] S711 , extract features using an unsupervised learning method for the URL path of the AIP endpoint to be managed in the final group, and finally generate a corresponding API endpoint model, and proceed to S712 .

[0206] S712 , through the API endpoint storage unit 640 , delete the API endpoint model under the original summary, and update the new API endpoint model obtained in S711 and the API endpoint to be managed into the database.

[0207] S713: Create a new summary group for the API endpoint to be managed, and proceed to S714.

[0208] S714: For the new summary group, use a classification algorithm to compare the URL paths of the API endpoints in the group in pairs, find API endpoints with only unique different values ​​or all the same values, divide them into one group, obtain several final groups, and enter S715.

[0209] S715 , extract features using an unsupervised learning method for the URL path of the AIP endpoint to be managed in the final group, and finally generate a corresponding API endpoint model, and proceed to S716 .

[0210] S716, through the API endpoint storage unit 640, record the summary of the AIP endpoint to be managed, the API endpoint model and the AIP endpoint under the summary group, enter step 17, and end the process.

[0211] S717, end.

[0212] API endpoint management can be achieved through S701-S717. Host and Path information can be extracted from traffic data to form API endpoints to be managed in the form of URLs. Then, URL features are extracted from the API endpoints to be managed and, based on the URL features, the API endpoints to be managed are divided into target groups with the same URL features. Based on the URLs of the API endpoints in the target groups, the API endpoints in the target groups are secondary grouped to obtain at least one final group; wherein, the path level values ​​corresponding to the URLs of the API endpoints in the final group are different by no more than 1.

[0213] Therefore, firstly, the identification and management of API endpoints through passive traffic and unsupervised learning methods reduces the cost of manual import and configuration, while improving the efficiency of API endpoint identification and management.

[0214] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0215] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, API endpoints can be managed in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0216] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group to which it belongs can be quickly found. Further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large number or all API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance consumption and improves management efficiency.

[0217] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive.

[0218] The API endpoint storage unit 640 can also extract and store security elements of the API endpoint.

[0219] Through the API endpoint white model modeling unit 650 , a security baseline white model may be generated based on the stored security elements of the API endpoint and may be maintained regularly.

[0220] Through the detection unit 660, the security baseline white model can be deployed in the network to implement security detection and discover anomalies and unknown threats.

[0221] The following may introduce some rules and logics for merging API endpoints in some embodiments. It should be noted that the merging process in the API management phase in any of the above embodiments may partially or completely refer to the following logic.

[0222] The following text may use some non-common terms, so before explaining the merge endpoint logic, we first explain these terms. Take http: / / www.example.com:8080 / api / demo / 123 / a1b2c3 / 964b0e94-96db-409f-b355-55fa98efa628 / login?username=admin&password=123456 as an example URL.

[0223] Word segmentation: "api", "demo", "username", etc. in the demonstration URL are word segmentations separated by specific delimiters.

[0224] Word type: For the word split in the demo URL, there are the following types: integer Integer (123), string with numbers AlphanumericString (a1b2c3) (Note: We agree that a string with a length greater than or equal to 6, at least 3 numbers and at least 1 letter is an AlphanumericString), UUID (964b0e94-96db-409f-b355-55fa98efa628), STRING (demo)

[0225] Payload similarity: Taking the REST-style API request payload as an example, in the two requests to be merged, the request parameters are shown in Table 1 below. Both requests have five request parameters, the same request parameters param1-param4, a unique parameter param5 for request 1, and a unique parameter param6 for request 2. The similarity between the two requests (number of common request parameters / max(number of parameters in request 1, number of parameters in request 2)) is 80%.

[0226]

[0227] The core logic of API endpoint merging:

[0228] The HTTP request protocol, HTTP request method, HTTP request header, and HTTP response header do not affect the merge result and are only used for business presentation.

[0229] Static files do not contain meaningful API endpoints and are not involved in API identification and merging.

[0230] The HOST, PATH, and request parameters contained in the URL can be split and extracted. In particular, if the request parameters are concatenated with "?" and "&" in the URL path, then these request parameters will be parsed and classified as part of the HTTP request payload as a factor in determining whether the API endpoint can be merged.

[0231] If the response status codes of the requests to be merged are different and are not in the range [200,300)∪{302}, these requests are considered to have failed, and the request data will be discarded and not included in the merge.

[0232] If the URL paths to be merged are exactly the same and the request methods are the same, regardless of whether their request payloads and response payloads are the same, they will be considered to be the same API, and the union of the request payloads will be considered the request payload of the API endpoint.

[0233] If the URLs to be merged satisfy the requirement that the URL path segmentation values ​​at the same position are different between any two URLs, but the type (INTEGER, UUID, AlphanumericString) is the same, and the segmentation values ​​at other positions are the same, the similarity of the request and response payloads will not be considered during the merge. As long as there are more than two URLs, these URLs will be merged into the same URL template, and the request payload union will be considered as the request payload of the API endpoint.

[0234] If the URLs in the set to be merged meet the requirement that the segmentation values ​​of the URL path segmentation in the same position are different but the type is the same (STRING), and the segmentation in other positions is the same, the URLs will be merged only if the number of URLs to be merged exceeds the threshold and the similarity between the request payload and the response payload is greater than a certain threshold (for example, 80% or 90%).

[0235] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0236] In this application, "and / or" means at least one of the two. The various embodiments in this application are described in a progressive manner. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the data processing device embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant details, refer to the description of the method embodiment.

[0237] Although the present application includes many specific implementation details, these should not be interpreted as limiting the scope of any disclosure or the scope of protection claimed, but are mainly used to describe the features of the specific disclosed embodiments. Certain features described in multiple embodiments of the present application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented in any suitable sub-combination. In addition, although features can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0238] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0239] The above are merely preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An API endpoint management method based on unsupervised learning, characterized in that: The method comprises: Get traffic data; Extracting Host and Path information from the traffic data and composing the API endpoint to be managed in the form of a URL; Extracting URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; the path abstract features are regular expressions of features of each path level; Generate summary information of the API endpoint to be managed based on the URL features; divide the API endpoint to be managed into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoint to be managed; According to the URLs of the API endpoints in the target group, the API endpoints in the target group are secondary grouped to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different path level values ​​of no more than a first preset number.

2. The API endpoint management method based on unsupervised learning according to claim 1, characterized in that: The method further comprises: For each final group, generate an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group; The extracting URL features from the API endpoint to be managed includes: In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

3. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: extracting security elements associated with the API endpoint to be managed from the traffic data; A security baseline white model is updated based on the security elements; the security baseline white model is used to perform security detection on traffic.

4. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: Get API interface documentation; The API endpoints in each of the final groups are compared with the endpoints in the API interface document, and the API endpoints that do not exist in the API interface document are determined as abnormal API endpoints.

5. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: After the API endpoint model corresponding to the final group is generated, the original API endpoint model of the final group is replaced, and the original API endpoint model is saved to perform model fallback in the event that the newly generated API model does not match accurately.

6. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The extracting URL features from the API endpoint to be managed includes: Splitting the URL of the API endpoint to be managed from left to right using a preset identifier to obtain at least one splitting feature; The first obtained split feature is determined as the agreement feature; The second obtained split feature is determined as the host feature; The number of remaining split features is determined as path-level features; For each of the remaining split features, the abstracted regular expression is determined as the path abstraction feature of the corresponding path level.

7. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The secondary grouping of the API endpoints in the target group according to the URLs of the API endpoints in the target group to obtain at least one final group includes: For each API endpoint in the target group, compare its URL with the URLs of other API endpoints path by path level to determine a first API endpoint with the same value at each path level or a second API endpoint with only one path level value different. The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

8. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: Generating an API endpoint model corresponding to the final group according to the URL and URL features of the API endpoints in the final group includes: The different values ​​in the URLs of the API endpoints in the final group are replaced with corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

9. An API endpoint management system based on unsupervised learning, characterized in that: The system comprises: A flow collection unit, used to obtain flow data; An API endpoint parsing unit, configured to extract Host and Path information from the traffic data and compose an API endpoint to be managed in the form of a URL; An API endpoint merging unit is configured to extract URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; the path abstract features are regular expressions of features of each path level; And, generating summary information of the API endpoint to be managed according to the URL characteristics; dividing the API endpoint to be managed into target groups; the API endpoints in the target group have the same summary information as the summary information of the API endpoint to be managed; and, performing secondary grouping of the API endpoints in the target group according to the URLs of the API endpoints in the target group to obtain at least one final group; wherein the URLs corresponding to the API endpoints in the final group differ in path level values ​​by no more than a first preset number; And, for each final group, generating an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group; API endpoint storage unit, used to store the URL of the API endpoint; API endpoint white model modeling unit, used to generate a security baseline white model; A detection unit is used to perform security detection on traffic based on the security baseline white model.

Images