API endpoint management method and system based on unsupervised learning

Abstract

The invention relates to the technical field of network security, in particular to an API endpoint management method and system based on unsupervised learning. According to the method, Host and Path information can be extracted from passive traffic to form to-be-managed API endpoints in a URL form, and two-stage grouping can be performed by extracting related URL features to realize unsupervised management of incremental APIs.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and specifically to an API endpoint management method and system based on unsupervised learning. Background Art

[0002] In today's era of rapid development of Internet technology, the communication between various applications and systems is becoming more frequent and close. Among them, API (Application Programming Interface) plays a vital role as a bridge for interaction between different applications. It can integrate the functions of different applications, allowing developers to easily call the functions of other applications in their own applications. For example, by connecting to the API of a third-party payment platform in a website, the payment function can be easily realized. This enables countless systems, platforms and applications to work together more efficiently, greatly improving the efficiency of various businesses in data processing, transmission, analysis and other aspects.

[0003] With the widespread use of APIs, their security issues are becoming increasingly prominent. As a specific function in an API service, an API endpoint is usually represented by a URL path of an HTTP request. An HTTP request is a request message sent by a browser to an HTTP server. Each HTTP request contains a request line, some header fields, and an optional data entity. In actual applications, there are a large number of API endpoints, and they will continue to increase with the development of the business. Accurately identifying and managing API endpoints has become the key to ensuring system security.

[0004] However, traditional API endpoint identification methods have many problems: First, the labor cost is high. Traditional methods rely on manual configuration and management, and when the source code cannot be obtained, endpoints are discovered through active scanning. This not only consumes a lot of manpower, material resources and time, but as the number of API endpoints increases, manual processing is difficult to ensure accuracy and real-time performance.

[0005] Second, it cannot automatically adapt to changes. In the face of dynamic changes in API endpoints, existing technologies require manual updates of configuration and management policies, which are prone to false positives and false negatives, affecting identification effects and security.

[0006] Third, the recognition accuracy is insufficient. Existing technologies are easily affected by factors such as data noise and insufficient sampling, resulting in poor recognition accuracy, making it difficult for security teams to detect and locate potential safety hazards in a timely manner.

[0007] Fourth, the management of endpoints is inefficient. When traditional methods process large amounts of data, the similarity comparison, classification, and merging of API endpoints takes a long time, and cannot achieve real-time, sustainable, and incremental identification. Each time, a large number of URLs are imported and merged repeatedly, which is extremely inefficient.

[0008] Fifth, active scanning findings are incomplete. Without a high-privilege management account, the API endpoint results discovered by active scanning are usually incomplete. Summary of the invention

[0009] In view of these problems in the existing technology, it is of great practical significance to study an API endpoint identification method and system based on unsupervised learning. As a machine learning method, unsupervised learning can automatically discover hidden patterns and structures in data without prior label information, providing new ideas and directions for solving the problem of API endpoint identification.

[0010] The present application discloses an API endpoint management method based on unsupervised learning. The method may include: obtaining traffic data; extracting Host and Path information from the traffic data, and forming an API endpoint to be managed in the form of a URL; extracting URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; dividing the API endpoint to be managed into target groups according to the URL features; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoint to be managed; according to the URLs of the API endpoints in the target group, performing secondary grouping on the API endpoints in the target group to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are not greater than a first preset number.

[0011] In some embodiments, the method also includes: for each final group, generating an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group; extracting URL features from the API endpoint to be managed includes: extracting URL features from the API endpoint to be managed when the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint.

[0012] In some embodiments, the method further includes: extracting security elements associated with the API endpoint to be managed from the traffic data; updating a security baseline white model based on the security elements; and using the security baseline white model to perform security detection on the traffic.

[0013] In some embodiments, the method further includes: obtaining an API interface document; comparing the API endpoints in each of the final groups with the endpoints in the API interface document, and determining the API endpoints that do not exist in the API interface document as abnormal API endpoints.

[0014] In some embodiments, the method further includes: after generating the API endpoint model corresponding to the final group, replacing the original API endpoint model of the final group, and saving the original API endpoint model to perform model rollback in the event that the newly generated API model matches inaccurately.

[0015] In some embodiments, the extracting of URL features from the API endpoint to be managed includes: using a preset identifier to split the URL of the API endpoint to be managed from left to right to obtain at least one split feature; determining the first split feature obtained as a protocol feature; determining the second split feature obtained as a host feature; determining the remaining number of split features as path level features; and for each of the remaining split features, determining the abstracted regular expression as a path abstract feature of the corresponding path level.

[0016] In some embodiments, the dividing the API endpoints to be managed into target groups according to the URL characteristics includes: generating summary information of the API endpoints to be managed according to the URL characteristics; dividing the API endpoints to be managed into target groups; the summary information possessed by the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0017] In some embodiments, the API endpoints in the target group are secondary grouped according to the path abstract features of each path level of the API endpoints in the target group to obtain at least one final group, including: for each API endpoint in the target group, its URL is compared with the URLs of other API endpoints path level by path level, and a first API endpoint whose values ​​at each path level are exactly the same or a second API endpoint whose value is different only at one path level is determined, and the API endpoints, the first API endpoint, and the second API endpoint are divided into a final group.

[0018] In some embodiments, generating an API endpoint model corresponding to the final group based on the URL and URL features of the API endpoint in the final group includes: replacing different values ​​in the URL of the API endpoint in the final group with corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

[0019] This application also proposes an API endpoint management system based on unsupervised learning. The system includes: a traffic collection unit, used to obtain traffic data; an API endpoint parsing unit, used to extract Host and Path information from the traffic data and form an API endpoint to be managed in the form of a URL; an API endpoint merging unit, used to extract URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; and, according to the URL features, the API endpoint to be managed is divided into target groups; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoint to be managed; and, according to the URLs of the API endpoints in the target group, the API endpoints in the target group are secondary grouped to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are not greater than a first preset number; and, for each final group, according to the URLs of the API endpoints in the final group, an API endpoint model corresponding to the final group is generated; an API endpoint storage unit, used to store the URLs of the API endpoints; an API endpoint white model modeling unit, used to generate a security baseline white model; and a detection unit, used to perform security detection on the traffic based on the security baseline white model.

[0020] In the scheme recorded in the aforementioned embodiment, Host and Path information can be extracted from traffic data to form an API endpoint to be managed in the form of a URL, and then URL features are extracted from the API endpoint to be managed and, based on the URL features, the API endpoint to be managed is divided into target groups with the same URL features, and based on the URLs of the API endpoints in the target group, the API endpoints in the target group are secondary grouped to obtain at least one final group; wherein, the values ​​of the path levels corresponding to the URLs of the API endpoints in the final group are different by no more than a first preset number.

[0021] Therefore, firstly, the API endpoint identification and management are performed through passive traffic and unsupervised learning, which reduces the cost of manual import and configuration, and improves the efficiency of API endpoint identification and management.

[0022] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0023] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, we can manage API endpoints in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0024] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group it belongs to can be quickly found, and then further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large or full amount of API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance and improves management efficiency.

[0025] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.

[0027] Figure 1 A flowchart of an API endpoint management method based on unsupervised learning is shown in an embodiment of the present application.

[0028] Figure 2 A flowchart of a method for extracting URL features illustrated in this application.

[0029] Figure 3 The figure is a flow chart of a method for establishing a security baseline white model shown in the present application.

[0030] Figure 4 This is a flow chart of the method for identifying abnormal API endpoints illustrated in this application.

[0031] Figure 5 The figure is a flowchart of the method for grouping by abstracts illustrated in the present application.

[0032] Figure 6 This is a schematic diagram of the structure of the API endpoint management system illustrated in this application.

[0033] Figure 7 This is a flowchart of the API endpoint management method illustrated in this application. DETAILED DESCRIPTION

[0034] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0035] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0036] In view of the problems existing in the existing technology, it is of great practical significance to study an API endpoint identification method and system based on unsupervised learning. As a machine learning method, unsupervised learning can automatically discover hidden patterns and structures in data without prior label information, providing new ideas and directions for solving the problem of API endpoint identification.

[0037] This application proposes an API endpoint management method based on unsupervised learning. The method can extract Host and Path information from passive traffic to form a URL-based API endpoint to be managed, and can extract relevant URL features to perform two-level grouping to achieve unsupervised management of incremental APIs.

[0038] The following is an example description with reference to the accompanying drawings. Figure 1 , Figure 1 A flowchart of an API endpoint management method based on unsupervised learning is shown in an embodiment of the present application.

[0039] Figure 1 The API endpoint management method based on unsupervised learning shown can be applied to an electronic device. The electronic device can execute the method by carrying software logic corresponding to the API endpoint management method based on unsupervised learning. The type of the electronic device can be a laptop, a computer, a server, a mobile phone, a PDA (Personal Digital Assistant, PDA), etc. The type of the electronic device is not particularly limited in this application. The electronic device can also be a client device or a server device.

[0040] like Figure 1 As shown, the method may include S102-S110. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0041] S102, obtaining traffic data.

[0042] The traffic may include request traffic, response traffic, and HTTP / HTTPS traffic. Request traffic is generated when a client sends a request to a server, and response traffic is generated when a server responds to a request. HTTP / HTTPS traffic is traffic of commonly used protocols for network communication.

[0043] When obtaining traffic data, for HTTP traffic, you can use a packet capture tool or probe such as Wireshark to directly capture packets on the network interface and extract traffic data. For example, in an enterprise internal network, install Wireshark on the gateway connecting the internal network and the external network, set the filtering conditions, and you can capture HTTP traffic.

[0044] For HTTPS traffic, due to its encryption characteristics, you need to first configure an SSL decryption device on the client or server side, and then use a packet capture tool to extract data after legal decryption.

[0045] Passive traffic, mirrored traffic, and non-active scanning traffic are mainly obtained by configuring the port mirroring function on network devices (such as switches). For example, in a large network, by mirroring the port traffic connected to important servers on the switch to electronic devices, these traffic data can be obtained for subsequent identification and discovery of AIP endpoints.

[0046] S104, extracting Host and Path information from the traffic data and composing an API endpoint to be managed in the form of a URL.

[0047] In an HTTP request, the Host information is used to identify the domain name or IP address of the target server. It is an important part of the URL and determines which server the request will be sent to. For example, for the request "https: / / www.example.com / api / data", "www.example.com" is the Host information. The Host information enables the client to accurately find the corresponding server and ensure that the target of data transmission is clear.

[0048] Path information refers to the path of specific resources on the target server. Taking the above request as an example, " / api / data" is the Path information, which indicates the specific resource location to be accessed on the server "www.example.com". The Path can be used to locate specific files, directories or API interfaces on the server.

[0049] Extract Host and Path information from traffic data and form the API endpoint to be managed in the form of a URL. The specific steps are as follows: First, parse the HTTP request message from the captured traffic data. In the message, the Host information is usually located in the request header field, starting with "Host:", followed by the server's domain name or IP address. For example, in the following request header: GET / api / data HTTP / 1.1; Host: www.example.com; User-Agent: Mozilla / 5.0; You can easily extract "www.example.com" as the Host information.

[0050] The Path information usually follows the HTTP request method (such as GET, POST, etc.). In the above example, the " / api / data" following "GET" is the Path information.

[0051] After extracting the Host and Path information, compose the URL in the format of "protocol: / / Host information: port number / Path information". If it is HTTP protocol, the default port number is 80; if it is HTTPS protocol, the default port number is 443. When using the default port number, the port number part can be omitted. If the "www.example.com" and " / api / data" extracted above are combined into a URL of "https: / / www.example.com / api / data", this URL can be used as the API endpoint to be managed for subsequent API endpoint identification, management, and security monitoring operations.

[0052] S106: Extracting URL features from the API endpoint to be managed.

[0053] The URL feature refers to the feature extracted based on the URL of the API endpoint to be managed, which can indicate the specific features of the endpoint to facilitate subsequent group management. The URL feature includes protocol features, host features, path level features, and path abstract features of each path level.

[0054] The present application may adopt various methods to extract the aforementioned ULR features, for example, displaying the URL and splitting it manually, but this method is inefficient.

[0055] In some embodiments, an automatic feature extraction method may be used. Figure 2 , Figure 2 FIG. 1 is a flow chart of a method for extracting URL features according to the present application. The method is an implementation method of S106. Figure 2 As shown, the method may include S202 - S210 .

[0056] S202, splitting the URL of the API endpoint to be managed from left to right using a preset identifier to obtain at least one splitting feature; S204, determining the first obtained split feature as a protocol feature; S206, determining the second obtained split feature as a host feature; S208, determining the remaining number of split features as path-level features; S210 , for each of the remaining split features, determine the abstracted regular expression as a path abstract feature of the corresponding path level.

[0057] Taking the URL of the API endpoint to be managed, "http: / / www.example.com / api / v1 / login", as an example, the steps to extract its URL features are as follows: S202, split the URL using a preset identifier: The preset identifier is generally ": / / " and " / ". First, split the URL from left to right based on ": / / " to divide it into two parts: "http" and "www.example.com / api / v1 / login"; then further split "www.example.com / api / v1 / login" based on " / " to obtain the split features of "www.example.com", "api", "v1", and "login".

[0058] S204, determine the protocol feature: the first split feature obtained is the protocol feature, so the protocol feature of this URL is "http", which indicates that the data transmission adopts the Hypertext Transfer Protocol.

[0059] S206, determining a host feature: The second split feature obtained is a host feature, namely "www.example.com", which represents the address of a server providing a service. The corresponding server can be located through the host name to obtain related API services.

[0060] S208, determine the path level characteristics: After determining the protocol and host characteristics, the remaining split characteristics are "api", "v1", and "login", a total of 3, so the path level characteristic is 3, which reflects the complexity of the API endpoint in the path structure and the hierarchical division of functional organization.

[0061] S210, determining the path abstract feature: For "api", it can be abstracted into a regular expression "[az]", with the number of characters being 3, indicating that this level is composed of 3 lowercase letters.

[0062] For "v1", it can be abstracted as "[az] [0-9]", and the number of characters is 2, indicating that this level consists of 1 lowercase letter and 1 number.

[0063] For "login", it can be abstracted as "[az]", and the number of characters is 5, indicating that this level is composed of 5 lowercase letters.

[0064] Thus, through S202-S210, automatic URL feature extraction can be achieved, which is convenient for subsequent endpoint management. In some methods, feature extraction algorithms such as neural networks can also be used for path abstract features.

[0065] S108: Divide the API endpoints to be managed into target groups according to the URL features.

[0066] This step is the first grouping of API endpoints. The URL features of the API endpoints in the target group are the same as the URL features of the API endpoints to be managed.

[0067] In this step, a URL feature matching method may be used, for example, URL features may be combined into vectors, and target groups may be found through Mahalanobis distance, cosine distance, and the like.

[0068] It is understandable that in this step, if a target group with the same characteristics as the URL can be matched, the endpoint to be managed can be placed in the target group. If no target group is matched, a group can be created for the API endpoint to be managed.

[0069] S110, performing secondary grouping on the API endpoints in the target group according to the URLs of the API endpoints in the target group to obtain at least one final group.

[0070] The URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are no more than the first preset number. The first preset number is set according to demand. For example, it can be 1 or 2, etc. Assuming that the first preset number is 1, after the second grouping, every two URLs in each final group are all the same, or only one level of path abstract features are different.

[0071] Taking the first preset number as 1 as an example, in S110, for each API endpoint in the target group, its URL can be compared with the URLs of other API endpoints path by path level to determine the first API endpoint whose values ​​at each path level are exactly the same or the second API endpoint whose value at only one path level is different. The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

[0072] For example, suppose the target group contains the following API endpoint URLs: https: / / www.example.com / api / v1 / user / 100.

[0073] https: / / www.example.com / api / v1 / user / 200.

[0074] https: / / www.example.com / api / v1 / user / 300.

[0075] https: / / www.example.com / api / v1 / profile / 500.

[0076] https: / / www.example.com / api / v1 / profile / 600.

[0077] https: / / www.example.com / api / v1 / profile / 700.

[0078] We set the first preset number to 1, which means that after the second grouping, every two URLs in each final group are either completely identical or have only one level of path abstract features that are different.

[0079] In specific operations, for each API endpoint in the target group, its URL is compared with the URLs of other API endpoints path by path level.

[0080] Take "https: / / www.example.com / api / v1 / user / 100" as an example. Compared with "https: / / www.example.com / api / v1 / user / 200", their protocol features are both "https", the host features are both "www.example.com", and the path level features "api", "v1", and "user" are the same. Only the last level numbers "100" and "200" are different, that is, only one path level value is different, which meets the conditions.

[0081] Similarly, "https: / / www.example.com / api / v1 / user / 100" The only difference between this number and https: / / www.example.com / api / v1 / user / 300 is the last number, so it meets the requirement.

[0082] "https: / / www.example.com / api / v1 / user / 100" Compared with "https: / / www.example.com / api / v1 / profile / 500", the first three layers "https", "www.example.com", "api", and "v1" are the same, but the fourth layer "user" and "profile" are different, and the last layer "100" and "500" are also different. The values ​​of two path layers are different, which does not meet the conditions.

[0083] So, you can change "https: / / www.example.com / api / v1 / user / 100", "https: / / www.example.com / api / v1 / user / 200" and "https: / / www.example.com / api / v1 / user / 300" are grouped into one final group.

[0084] Change "https: / / www.example.com / api / v1 / profile / 500", "https: / / www.example.com / api / v1 / profile / 600", “https: / / www.example.com / api / v1 / profile / 700” is classified into one final group.

[0085] By continuously comparing and dividing all API endpoints in the target group in the above manner, at least one final group can be obtained, and the URLs of the API endpoints in each final group satisfy the condition that the values ​​of the path levels are different and not greater than a first preset number, thereby realizing group management of the API endpoints.

[0086] Through the scheme recorded in S102-S110, Host and Path information can be extracted from traffic data to form an API endpoint to be managed in the form of a URL, and then URL features are extracted from the API endpoint to be managed and divided into target groups with the same URL features based on the URL features, and the API endpoints in the target group are secondary grouped based on the URLs of the API endpoints in the target group to obtain at least one final group; wherein the values ​​of the path levels corresponding to the URLs of the API endpoints in the final group are different by no more than a first preset number.

[0087] Therefore, firstly, the API endpoint identification and management are performed through passive traffic and unsupervised learning, which reduces the cost of manual import and configuration, and improves the efficiency of API endpoint identification and management.

[0088] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0089] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, we can manage API endpoints in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0090] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group it belongs to can be quickly found, and then further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large or full amount of API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance and improves management efficiency.

[0091] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive.

[0092] In some embodiments, the API endpoints under the final grouping can be modeled. If a newly identified API endpoint matching the model is found, it can be released. If a newly identified API endpoint not matching the model is found, it can be analyzed in detail. If it is analyzed to be a risky access, it can be truncated and an alarm can be issued immediately. If it is found to be a safe access, it can be released and incremental management can be performed using solutions such as S106-S110.

[0093] In some embodiments, after S110, the method further includes: For each final group, an API endpoint model corresponding to the final group is generated according to the URL of the API endpoint in the final group.

[0094] In this step, different values ​​in the URL of the API endpoint in the final group may be replaced with corresponding regularized expressions to obtain an API endpoint model corresponding to the final group.

[0095] Take the final grouping divided above as an example. The URLs included in the first final grouping are: https: / / www.example.com / api / v1 / user / 100 https: / / www.example.com / api / v1 / user / 200 https: / / www.example.com / api / v1 / user / 300 In these URLs, the protocol "https", the host "www.example.com", and the first three layers of the path hierarchy "api", "v1", and "user" are all fixed, and only the numbers in the last layer are different. These different numbers represent different user IDs. In order to abstract this set of URLs into a common API endpoint model, the changing part (that is, the numbers in the last layer) can be replaced with the corresponding regularized expression. Since this part is a number, the regular expression "[0-9]" can be used, which means matching one or more numbers. Therefore, the API endpoint model corresponding to this final grouping is: https: / / www.example.com / api / v1 / user / [0-9]. Through this model, the common characteristics and changing patterns of this group of API endpoints can be accurately represented, where "[0-9]" covers all possible numerical values ​​that may appear at this path level.

[0096] The second final group contains the URLs: https: / / www.example.com / api / v1 / profile / 500 https: / / www.example.com / api / v1 / profile / 600 https: / / www.example.com / api / v1 / profile / 700 Similarly, the first three layers of the protocol, host, and path levels remain unchanged, and the numbers in the last layer are different, representing different profile identifiers. Replace these different values ​​with the regular expression "[0-9]", and the API endpoint model corresponding to the final group is obtained: https: / / www.example.com / api / v1 / profile / [0-9]. This model can summarize the commonalities of this group of API endpoints.

[0097] By replacing different values ​​in the URLs of the API endpoints in the final group with corresponding regularized expressions to generate an API endpoint model, the characteristics of the API endpoints in the final group can be described in a concise and general way, which plays an important role in API management, analysis, and security monitoring.

[0098] In S106, extracting URL features from the API endpoint to be managed includes: In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

[0099] For example, taking the aforementioned API endpoint related content as an example, assume that there are already generated API endpoint models, such as "https: / / www.example.com / api / v1 / user / [0-9]" and "https: / / www.example.com / api / v1 / profile / [0-9]".

[0100] When a new API endpoint to be managed appears, the first thing to do is to determine whether it matches any existing API endpoint model. For example, if the new API endpoint to be managed is "https: / / www.example.com / api / v2 / product / 123", compare it with the existing model. From the protocol feature, both are "https", and the host feature is also "www.example.com", but in the path level feature, the version number "v2" after "api" is different from the "v1" in the existing model, and "product" is different from the "user" and "profile" in the existing model, so it does not match any existing API endpoint model.

[0101] Furthermore, relevant technologies can be used to determine whether the API endpoint to be managed is an abnormal endpoint. Abnormal endpoints may be some malicious request endpoints with format errors, non-compliance with basic network communication specifications, or unknown sources. If the "https: / / www.example.com / api / v2 / product / 123" endpoint is an abnormal endpoint after inspection, the relevant access can be intercepted and an alarm can be issued. If it is not an abnormal endpoint, the URL features can be extracted from the API endpoint to be managed for subsequent endpoint management.

[0102] Therefore, the API endpoints in the final group can be modeled to obtain a model that can characterize the features and implement risk detection.

[0103] In some embodiments, after the API endpoint model corresponding to the final group is generated, the original API endpoint model of the final group is replaced, and the original API endpoint model is saved to perform model rollback in the event that the newly generated API model does not match accurately.

[0104] In the API endpoint management process, after the API endpoint model corresponding to a specific final group is generated, model replacement and retention operations need to be performed.

[0105] Specifically, the newly generated API endpoint model can replace the original API endpoint model of the final group. This can make the management and analysis of API endpoints more accurate and efficient. The new model is usually based on more comprehensive data and in-depth analysis, and can more accurately reflect the actual status of the API endpoint.

[0106] In actual application scenarios, the newly generated API model may have mismatches. For example, in some special situations, the new model cannot accurately identify the API endpoint, or the analysis of traffic data may be biased. Once such a situation occurs, the model rollback operation can be performed to re-apply the original API endpoint model retained in the early stage. By rolling back to the original model, it can ensure that the management and analysis of the API endpoint can continue normally, avoiding misjudgments and security risks caused by the inaccuracy of the new model.

[0107] In some embodiments, a security baseline white model can also be established based on the security elements of the API endpoint to perform security testing. Figure 3 , Figure 3 FIG. 1 is a flow chart of a method for establishing a security baseline white model in this application. Figure 3 As shown, it may include S302 - S304 .

[0108] S302: extracting security elements associated with the API endpoint to be managed from the traffic data.

[0109] S304, updating a security baseline white model based on the security elements; the security baseline white model is used to perform security detection on traffic.

[0110] The security elements may be elements extracted from HTTP messages such as request URLs, request messages, and response messages, for example, API transmission data format, API transmission data size, API transmission delay, number of API parameter names, number of API paths, API parameter values, etc. These elements may be standardized and combined with some preset rules to form a security baseline white model. The security baseline white model may be an artificial intelligence model, including a model constructed by linear regression, neural network, deep learning, or a large model.

[0111] The security baseline white model can include the indicator ranges of normal traffic. For example, under normal circumstances, the size of API transmission data may be between 1KB and 50KB, and the API transmission delay may be between 100 milliseconds and 300 milliseconds. When the relevant indicators in the traffic data exceed these ranges, it may be risky traffic, such as the data size suddenly soaring to 100KB, or the transmission delay extending to 1000 milliseconds. These abnormal situations may indicate that the network is under attack, such as malicious injection of large amounts of data, denial of service attacks resulting in delayed responses, etc.

[0112] Assume we have an API endpoint to be managed, "https: / / www.example.com / api / v1 / product / query", and the traffic data associated with it: API transmission data format: If the request message is {"category":"electronics","priceRange":[100, 500]}, it can be determined that the data format is JSON.

[0113] API transmission data size: After calculation, the request message size is 40 bytes.

[0114] API transmission latency: It takes 250 milliseconds from sending a request to receiving a response.

[0115] Number of API parameter names: There are two parameter names in the message: "category" and "priceRange".

[0116] Number of API paths: For the request URL, the path level is 4 ("api" "v1" "product" "query").

[0117] API parameter values: For example, the value of “category” is “electronics” and the value of “priceRange” is “[100, 500]”.

[0118] After obtaining the relevant security factors, the security baseline white model can be updated by methods such as linear regression. For example, taking the size of API transmission data as an example, the data size in a large amount of historical traffic data and the corresponding normal or not marks are used as a training set to build a model using a linear regression algorithm. Assuming that the current extracted API transmission data size is 40 bytes, this data is included in the model for calculation, and 1KB - 60KB can be obtained (hypothetical value, which will actually be calculated based on the algorithm and data). By continuously incorporating newly extracted security factors into the linear regression model for calculation and updating, the security baseline white model can accurately reflect the characteristics of normal traffic, thereby effectively performing security detection on subsequent traffic and promptly discovering potential security risks.

[0119] In some embodiments, abnormal endpoints can also be identified among the managed API endpoints, including zombie API endpoints and shadow API endpoints. Figure 4 , Figure 4 This is a flow chart of the method for identifying abnormal API endpoints illustrated in this application.

[0120] S402, obtain API interface documentation.

[0121] S404: Compare the API endpoints in each of the final groups with the endpoints in the API interface document, and determine the API endpoints that do not exist in the API interface document as abnormal API endpoints.

[0122] The interface document refers to the recorded known API endpoints, which can be maintained manually or by the system.

[0123] In an embodiment, the URL of the API endpoint in each final group can be compared with the URL of the endpoint in the obtained API interface document. Assuming that the URL is an endpoint of https: / / www.example.com / api / v1 / custom / operation, no matching endpoint record is searched in the API interface document, indicating that the endpoint is an abnormal endpoint, and an alarm is issued. Normal endpoints can include zombie API endpoints and shadow API endpoints.

[0124] Identifying these abnormal API endpoints can effectively prevent potential security risks.

[0125] In some embodiments, in S108 , a method of calculating a summary may be used to perform the first step of grouping, thereby reducing the amount of calculation, improving the grouping efficiency, and further improving the efficiency of API endpoint management.

[0126] See also Figure 5 , Figure 5 The figure is a flowchart of the method for grouping by abstracts illustrated in the present application.

[0127] S502: Generate summary information of the API endpoint to be managed according to the URL feature.

[0128] S504, dividing the API endpoints to be managed into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0129] The method for calculating summary information can refer to the summary algorithm in related calculations, such as MD5, SHA-1, etc.

[0130] Take http: / / www.example.com / api / v1 / login as an example. Figure 2The URL features extracted by the illustrated method may include: the protocol feature is "http", the host feature is "www.example.com", and the path abstract features are "[az]" (3 lowercase letters), "[az][0-9]" (1 lowercase letter and 1 number), and "[az]" (5 lowercase letters). The summary information can be calculated through MD5. The first step of grouping can be completed using the summary information.

[0131] This application also proposes an API endpoint management system based on unsupervised learning. Figure 6 , Figure 6 This is a schematic diagram of the structure of the API endpoint management system illustrated in this application. Figure 6 As shown, the system 600 may include: A flow collection unit 610, used to obtain flow data; An API endpoint parsing unit 620 is used to extract Host and Path information from the traffic data and compose an API endpoint to be managed in the form of a URL; The API endpoint merging unit 630 is used to extract URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; And, according to the URL features, the API endpoints to be managed are divided into target groups; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoints to be managed; and, according to the URLs of the API endpoints in the target group, performing secondary grouping on the API endpoints in the target group to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are no greater than a first preset number; And, for each final group, generating an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group; The API endpoint storage unit 640 is used to store the URL of the API endpoint; API endpoint white model modeling unit 650, used to generate a security baseline white model; The detection unit 660 is configured to perform security detection on traffic based on the security baseline white model.

[0132] For the relevant description of the steps executed by each unit of the system, reference may be made to any of the previous method embodiments.

[0133] In some embodiments, the API endpoint merging unit 630 further: In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

[0134] In some embodiments, the API endpoint storage unit 640 further extracts security elements associated with the API endpoint to be managed from the traffic data; The API endpoint white model modeling unit 650 further updates the security baseline white model based on the security elements; the security baseline white model is used to perform security detection on the traffic.

[0135] In some embodiments, the detection unit 660 further obtains an API interface document; The API endpoints in each of the final groups are compared with the endpoints in the API interface document, and the API endpoints that do not exist in the API interface document are determined as abnormal API endpoints.

[0136] In some embodiments, the API endpoint storage unit 640 further replaces the original API endpoint model of the final group after generating the API endpoint model corresponding to the final group, and saves the original API endpoint model to perform model fallback in the event that the newly generated API model does not match accurately.

[0137] In some embodiments, the API endpoint merging unit 630 further: Using a preset identifier, the URL of the API endpoint to be managed is split from left to right to obtain at least one split feature; The first obtained split feature is determined as the agreement feature; The second obtained split feature is determined as the host feature; The number of remaining split features is determined as path-level features; For each of the remaining split features, the abstracted regular expression is determined as the path abstract feature of the corresponding path level.

[0138] In some embodiments, the API endpoint merging unit 630 further: Generate summary information of the API endpoint to be managed according to the URL features; The API endpoints to be managed are divided into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

[0139] In some embodiments, the API endpoint merging unit 630 further: For each API endpoint in the target group, compare its URL with the URLs of other API endpoints path by path level to determine a first API endpoint with exactly the same value at each path level or a second API endpoint with only one different value at one path level, The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

[0140] In some embodiments, the API endpoint merging unit 630 further: The different values ​​in the URL of the API endpoint in the final group are replaced with the corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

[0141] See also Figure 7 , Figure 7 This is a flowchart of the API endpoint management method illustrated in this application. Figure 7 The illustrated method can be based on Figure 6 Schematic API endpoint management system implementation. Figure 7 As shown, the method may include S701-S717.

[0142] S701, start.

[0143] S702, collect HTTP traffic through the traffic collection unit 610, and some HTTPS traffic that supports decryption can also be collected, and enter S703.

[0144] S703, through the API endpoint parsing unit 620, the request Host and path are extracted from the traffic and form a complete URL path to obtain the AIP endpoint to be managed, and enter S704.

[0145] S704 - S711 and S713 - S715 are executed by the API endpoint merging unit 630 .

[0146] S704, determine whether it is an API request based on the relevant information in the traffic, if it is, proceed to S705; if not, proceed to S717 and the process ends.

[0147] S705, split the URL path of the AIP endpoint to be managed by : / / or / as a unit, and extract URL features. The URL features include protocol features, host features, path level features, and path abstract features of each path level. Then proceed to S706.

[0148] S706, perform digest calculation based on the combined URL features obtained in the previous step to obtain a unique digest result of the AIP endpoint to be managed, and proceed to S707.

[0149] S707, determine whether it is incremental training, if yes, execute S708; if no, execute S713.

[0150] S708, determining whether the summary generated by the AIP endpoint to be managed already has a historical group, if yes, proceeding to S709; if no, proceeding to S713.

[0151] S709, storing the AIP endpoint to be managed into the target group with the same summary, and proceeding to S710.

[0152] S710, regroup the AIP endpoints to be managed in the target group, find out the AIP endpoints with only unique different values ​​or all the same values, and group them into one group to obtain at least one final group, so as to enter S711 for further processing.

[0153] S711, extract features using an unsupervised learning method for the URL path of the AIP endpoint to be managed under the final group, and finally generate a corresponding API endpoint model, and enter S712.

[0154] S712, through the API endpoint storage unit 640, delete the API endpoint model under the original summary, and update the new API endpoint model obtained in S711 and the API endpoint to be managed into the database.

[0155] S713, create a new summary group for the API endpoint to be managed, and proceed to S714.

[0156] S714, for the new summary grouping, use the classification algorithm to compare the URL paths of the API endpoints in the group in pairs, find API endpoints with only unique different values ​​or all the same values, divide them into one group, obtain several final groups, and enter S715.

[0157] S715, extract features using an unsupervised learning method for the URL path of the AIP endpoint to be managed in the final group, and finally generate a corresponding API endpoint model, and enter S716.

[0158] S716, record the summary of the AIP endpoint to be managed, the API endpoint model and the AIP endpoint under the summary group through the API endpoint storage unit 640, enter step 17, and end the process.

[0159] S717, end.

[0160] API endpoint management can be achieved through S701-S717. Host and Path information can be extracted from traffic data to form API endpoints to be managed in the form of URLs. URL features are then extracted from the API endpoints to be managed and, based on the URL features, the API endpoints to be managed are divided into target groups with the same URL features. Based on the URLs of the API endpoints in the target groups, the API endpoints in the target groups are secondary grouped to obtain at least one final group; wherein the path level values ​​corresponding to the URLs of the API endpoints in the final group are different by no more than 1.

[0161] Therefore, firstly, the API endpoint identification and management are performed through passive traffic and unsupervised learning, which reduces the cost of manual import and configuration, and improves the efficiency of API endpoint identification and management.

[0162] Second, by combining a two-level grouping algorithm with unsupervised learning, API endpoint identification and management can be more precise and generalized, reducing false positives and improving identification accuracy and management efficiency.

[0163] Third, by automating and continuously identifying API endpoints, we can ensure that even when facing newly launched business systems, we can manage API endpoints in a timely manner without the need for tedious steps such as manual import, configuration, and active scanning, thereby improving the timeliness of identification and management.

[0164] Fourth, by extracting URL features, the attributes of the URL path can be recorded. When a new URL path appears, the group it belongs to can be quickly found, and then further secondary grouping can be performed within the group to achieve unsupervised training, thereby supporting incremental management in the API endpoint identification process. There is no need to import a large or full amount of API endpoint URLs for unsupervised learning each time, which avoids a lot of time and performance and improves management efficiency.

[0165] Fifth, API endpoint identification is performed through passive traffic and unsupervised learning. Compared with the active scanning discovery method, the discovered API endpoints are more comprehensive.

[0166] The API endpoint storage unit 640 can also extract and store security elements of the API endpoint.

[0167] Through the API endpoint white model modeling unit 650 , a security baseline white model may be generated based on the stored security elements of the API endpoint and may be maintained regularly.

[0168] Through the detection unit 660, the security baseline white model can be deployed in the network to implement security detection and discover anomalies and unknown threats.

[0169] Some rules and logics for merging API endpoints in some embodiments may be introduced below. It should be noted that the merging process in the API management phase in any of the above embodiments may partially or completely refer to the following logic.

[0170] The following text may use some non-common terms, so before explaining the merge endpoint logic, first explain these terms, taking http: / / www.example.com:8080 / api / demo / 123 / a1b2c3 / 964b0e94-96db-409f-b355-55fa98efa628 / login?username=admin&password=123456 as an example URL.

[0171] Word segmentation: "api", "demo", "username" and so on in the demonstration URL are word segmentations separated by specific delimiters.

[0172] Word segmentation type: For the word segmentation in the demo URL, there are the following types: integer Integer (123), string with numbers AlphanumericString (a1b2c3) (Note: We agree that the string with a length greater than or equal to 6, at least 3 numbers and at least 1 letter is AlphanumericString), UUID (964b0e94-96db-409f-b355-55fa98efa628), STRING (demo) Payload similarity: Taking the REST-style API request payload as an example, in the two requests to be merged, the request parameters are shown in Table 1 below. There are 5 request parameters in both requests. Both requests have the same request parameters param1-param4, request 1 has a unique parameter param5, and request 2 has a unique parameter param6. The similarity of the two requests (the number of identical request parameters / max (the number of request 1 parameters, the number of request 2 parameters)) is 80%.

[0173] The core logic of API endpoint merging: HTTP request protocol, HTTP request method, HTTP request header, and HTTP response header do not affect the merging result and are only used for business display.

[0174] Static files do not contain meaningful API endpoints and do not participate in API identification and merging.

[0175] The HOST, PATH, and request parameters contained in the URL can be split and extracted. In particular, if the request parameters are concatenated with "?" and "&" on the URL path, these request parameters will be parsed and classified as part of the HTTP request payload as a factor in determining whether the API endpoint can be merged.

[0176] If the response status codes of the requests to be merged are different and are not in the range of [200,300)∪{302}, these requests are considered to have failed, and the request data will be discarded and will not be merged.

[0177] If the URL paths to be merged are exactly the same and the request methods are the same, regardless of whether their request payloads and response payloads are the same, they will be considered to be the same API, and the union of the request payloads will be considered to be the request payload of the API endpoint.

[0178] If the URL set to be merged satisfies that the segmentation values ​​of the same position of the URL path segmentation between the two are different, but the type (INTEGER, UUID, AlphanumericString) is the same, and the segmentation in other positions is the same, the similarity of the request and response payloads will not be considered when merging. As long as there are more than two URLs, these URLs will be merged into the same URL template, and the request payload is taken as the union and is considered to be the request payload of the API endpoint.

[0179] If in the set of URLs to be merged, the URL path segmentation between each pair has different segmentation values ​​at the same position but the same type (STRING), and the segmentation at other positions is the same, the URLs will be merged only if the number of URLs to be merged exceeds the threshold and the similarity between the request payload and the response payload is greater than a certain threshold (for example, 80% or 90%).

[0180] Those skilled in the art should understand that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] "And / or" in this application means at least one of the two. The various embodiments in this application are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0183] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0184] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An API endpoint management method based on unsupervised learning, characterized in that: The method comprises: Get traffic data; Extracting Host and Path information from the traffic data and composing a to-be-managed API endpoint in the form of a URL; Extracting URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; According to the URL features, the API endpoints to be managed are divided into target groups; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoints to be managed; According to the URLs of the API endpoints in the target group, the API endpoints in the target group are secondary grouped to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different path level values ​​that are no more than a first preset number.

2. The API endpoint management method based on unsupervised learning according to claim 1, characterized in that: The method further comprises: For each final group, according to the URL of the API endpoint in the final group, generate an API endpoint model corresponding to the final group; The step of extracting URL features from the API endpoint to be managed includes: In a case where the API endpoint to be managed does not match any existing API endpoint model and the API endpoint to be managed is not an abnormal endpoint, a URL feature is extracted from the API endpoint to be managed.

3. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: Extracting security elements associated with the API endpoint to be managed from the traffic data; The security baseline white model is updated based on the security elements; the security baseline white model is used to perform security detection on traffic.

4. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: Get API interface documentation; The API endpoints in each of the final groups are compared with the endpoints in the API interface document, and the API endpoints that do not exist in the API interface document are determined as abnormal API endpoints.

5. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The method further comprises: After the API endpoint model corresponding to the final group is generated, the original API endpoint model of the final group is replaced, and the original API endpoint model is saved to perform model rollback in the event that the newly generated API model does not match accurately.

6. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The step of extracting URL features from the API endpoint to be managed includes: Using a preset identifier, the URL of the API endpoint to be managed is split from left to right to obtain at least one split feature; The first obtained split feature is determined as the agreement feature; The second obtained split feature is determined as the host feature; The number of remaining split features is determined as path-level features; For each of the remaining split features, the abstracted regular expression is determined as the path abstract feature of the corresponding path level.

7. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The dividing the API endpoints to be managed into target groups according to the URL features includes: Generate summary information of the API endpoint to be managed according to the URL features; The API endpoints to be managed are divided into target groups; the summary information of the API endpoints in the target group is the same as the summary information of the API endpoints to be managed.

8. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The second grouping of the API endpoints in the target group according to the path abstraction feature of each path level of the API endpoints in the target group to obtain at least one final group includes: For each API endpoint in the target group, compare its URL with the URLs of other API endpoints path by path level to determine a first API endpoint with exactly the same value at each path level or a second API endpoint with only one different value at one path level, The API endpoint, the first API endpoint, and the second API endpoint are divided into a final group.

9. The API endpoint management method based on unsupervised learning according to claim 2, characterized in that: The step of generating an API endpoint model corresponding to the final group according to the URL and URL features of the API endpoint in the final group includes: The different values ​​in the URL of the API endpoint in the final group are replaced with the corresponding regularized expressions to obtain the API endpoint model corresponding to the final group.

10. An API endpoint management system based on unsupervised learning, characterized in that: The system comprises: A flow collection unit, used for acquiring flow data; An API endpoint parsing unit, used to extract Host and Path information from the traffic data and compose an API endpoint to be managed in the form of a URL; An API endpoint merging unit, used to extract URL features from the API endpoint to be managed; the URL features include protocol features, host features, path level features, and path abstract features of each path level; And, according to the URL features, the API endpoints to be managed are divided into target groups; the URL features of the API endpoints in the target group are the same as the URL features of the API endpoints to be managed; and, according to the URLs of the API endpoints in the target group, performing secondary grouping on the API endpoints in the target group to obtain at least one final group; the URLs corresponding to the API endpoints in the final group have different values ​​of path levels that are no greater than a first preset number; And, for each final group, generating an API endpoint model corresponding to the final group according to the URL of the API endpoint in the final group; API endpoint storage unit, used to store the URL of the API endpoint; API endpoint white model modeling unit, used to generate a security baseline white model; A detection unit is used to perform security detection on traffic based on the security baseline white model.

Images