System and method for dynamically generating set of API endpoints

Abstract

A method and system for generating a set of API endpoints includes receiving a network data request to extract an original URL string and http Method therefrom; splitting the URL character string into component groups; and constructing a component tree. Dynamic components are detected and replaced with a universal identifier. The component tree is then folded by merging the same branches into a single branch, thereby providing a set of API endpoints. Detection of a dynamic component may include: determining that a number of child nodes paired with a parent node is relatively large; detecting that the occurrence frequency of the father node component in the preset time is relatively higher than the occurrence frequency of the corresponding child node component; detecting the relatively high similarity of the grandson node component sharing the father node; and detecting a sub-component having the randomly generated sequence of characters.

Description

[0001] This application is a divisional application with an application date of October 24, 2020, application number 202080032692.1, and invention name “System and method for dynamically generating a set of API endpoints”. The parent application is an application entering the Chinese national phase of international application number PCT / US2020 / 057240 submitted to the International Patent Office on October 24, 2020, claiming the benefit of U.S. Provisional Application No. 16 / 736,034 entitled “SYSTEM AND METHOD TO DYNAMICALLY GENERATEA SET OF API ENDPOINTS” filed on January 7, 2020. The above application is fully incorporated herein by reference. Technical Field

[0002] The present application relates generally to application program interfaces or "APIs" that allow two software programs to communicate with each other, and more particularly to a system and method for generating a set of API endpoints associated with a web application based on network traffic directed to the web application. Prior art

[0003] An application program interface, or "API," is simply computer software code that allows two software programs to communicate with each other. APIs work by sending requests for information from a web application or web server and receiving responses. An API endpoint is the point at which an API connects to a software program. In other words, an API endpoint is a specific digital location where a program can send a request for information and access the corresponding resource.

[0004] The performance of an API depends on its ability to successfully communicate with the API endpoint. API endpoints, usually defined as a unique tuple {URL, Method}, constitute the most basic command unit for most modern web applications. URL stands for Uniform Resource Locator, which is used to specify the location of a web resource on a computer network and the mechanism for retrieving the resource. A typical URL may have the form http: / / www.example.com / index.html, which indicates the protocol (http), the host name (www.example.com), and the file name (index.html). The "Method" component refers to common http method requests, such as POST, GET, PUT, PATCH, and DELETE.

[0005] Learning the set of API endpoints exposed by a web application simply from inspecting traffic has numerous applications such as data profiling and user behavior profiling in web application firewalls. Additionally, the learned set of API endpoints has great benefits as an app observability tool for system administrators. Once the set of API endpoints is learned, it can be used to profile and create other useful information about the application such as typical user behavior, etc. This can be presented as security and business intelligence insights consumed by business, security, and site reliability engineering teams.

[0006] For example, such information is valuable for security purposes such as: 1) "whitelisting" of known API endpoints; 2) profiling of known API endpoints; 3) user behavior analysis using API endpoint access sequences; and 4) denial of service (DoS) attack detection based on per-API endpoint access rates. In addition to security considerations, such information is also valuable for: 1) identifying the slowest API endpoints; 2) identifying those most prone to errors; and 3) capacity planning.

[0007] Layer 7 or L7 refers to the topmost layer in the seven-layer OSI model of the Internet; it is also known as the "application layer". The application layer is an abstract layer that specifies the shared communication protocols and interface methods used by hosts in a communication network. In the OSI model, the application layer serves as a user interface responsible for displaying received information to the user. As an explanation, the HTTP request and response used to load a web page is a layer 7 event.

[0008] To date, efforts to learn API endpoints have been limited to direct parent-child relationship checks on {URL,Method} tuples. One problem with this is that when matching URLs against a learned set of API endpoints, there may be dynamic components in the URLs that need to be mapped to the "any" or "*" operators of regular expressions ("regex"), i.e., sequences of characters that define a search pattern. Without the ability to automatically identify these dynamic components, any {URL,Method} system ends up learning an exhaustive set of all unique URLs. This approach results in learning duplicate URLs that point to the same API endpoint, leading to an unmanageably large set of URLs.

[0009] For example, the web application hosted at www.reddit.com uses a unique ID in the form of a seemingly random string of characters to identify unique objects. In the URL listed below: https: / / reddit.com / r / news / comments / 8zo9or The characters "8zo9or" represent this unique ID, or a random string. Static properties associated with this random string may appear later in the URL, as exemplified below: https: / / reddit.com / r / news / comments / 8zo9or / top / .

[0010] A web-based application may have thousands of such unique IDs embedded in its URL. However, if these thousands of unique IDs are viewed as thousands of different API endpoints, they do not really contribute any useful information; this is because the functionality performed at each such endpoint is the same. Summary of the invention

[0011] Therefore, an object of the present invention is to provide a system and method for generating a set of API endpoints associated with a web application by inspecting network data traffic directed to the web application.

[0012] Another object of the present invention is to provide such a system and method for generating a set of API endpoints, which can be generated by inspecting network data traffic in real time.

[0013] It is yet another object of the present invention to provide such a system and method for generating a set of API endpoints, which can optionally be generated by examining offline access logs of network data traffic generated by a network server.

[0014] It is yet another object of the present invention to provide such a system and method for generating a set of API endpoints that highly accurately distinguishes between static components and dynamic components.

[0015] It is another object of the present invention to provide such a system and method for generating a set of API endpoints that compress similar URLs that differ from each other only with respect to dynamic components.

[0016] These and other objects of the present invention will become more apparent to those skilled in the art as the description of the present invention proceeds.

[0017] Briefly, and according to its various embodiments, a first aspect of the present invention relates to a method for generating a set of API endpoints by inspecting network data traffic. This method includes receiving a plurality of network data requests, the plurality of network data requests being listed in a {URL, Method} format to identify command actions supported by a web application. The HTTP request header of such received network data requests is inspected to extract the original URL string therefrom. Each such original URL string is split into a plurality of components. An original component tree is assembled, the original component tree consisting of a plurality of branches formed from nodes, wherein each node is a component of at least one original URL string. This method analyzes each component in the original component tree to determine whether such component is static or dynamic. The original component tree is modified to replace components determined to be dynamic with a universal identifier. The original component tree is then collapsed to form a collapsed component tree by merging identical branches into a single branch. The API endpoint is then obtained from the collapsed component tree.

[0018] The received network data request may be received in real time from a data packet transmitted through a data network. Optionally, the received network data request may be received as an offline access log generated by a network server.

[0019] In various embodiments of this method, splitting each of the original URL strings into a set of components includes detecting slash characters " / " within the original URL string and obtaining the components from character groups separated by such slash characters.

[0020] In some embodiments of this method, the obtained set of API endpoints consists of a so-called "tuple" expressed in the form of {Collapsed URL,Method}, where "CollapsedURL" represents a URL string in which dynamic components are replaced by universal identifiers, and where "Method" represents an http method request, such as POST, GET, PUT, PATCH or DELETE.

[0021] In various embodiments of this method, the original component tree includes at least one (but typically more than one) "parent" node, which includes components located closer to the starting point of the original URL string. In addition, the original component tree includes at least one (but typically more than one) "child" node, which includes components located farther from the starting point of the original URL string than the parent node. By way of example, in the original URL string http: / / wafdemo.com / api / user / , the component "api" can be considered a "parent" node, while the component "user" can be considered a "child" node.

[0022] In some embodiments of this method, the process of determining whether a component is dynamic evaluates the number of "child" nodes paired with a particular "parent" node. In the case where the number of child nodes paired with a single parent node is relatively large, the child nodes are likely to be dynamic.

[0023] In some embodiments of this method, the process of determining whether a component is dynamic involves tracking the number of times a particular component appears in a received network data request within a predetermined time period. The number of occurrences of a component of a parent node within the predetermined time period is compared with the number of occurrences of a corresponding child node associated with the parent node within the predetermined time period. If the number of occurrences of a parent node component is relatively high, and the number of occurrences of each child node component is relatively low, then the child component is likely to be dynamic.

[0024] In some embodiments of this method, the process of determining whether a particular component is dynamic involves determining the similarity of the grandchild node components (relative to each other) of the child nodes that share the same parent node. In other words, focus on a particular parent node, find out its child nodes, and then for each such child node, form a set of corresponding grandchild components. For example, if a parent node A has 3 child components X, Y, and Z, and if each of the child components X, Y, and Z has its own child node component (that is, the grandchild node of the parent node A), then the logic element calculates the similarity between the sets of the child nodes of X, the child nodes of Y, and the child nodes of Z. If it is found that there is a relatively high similarity between the sets of these grandchild components, then the child node components X, Y, and Z may be dynamic. The calculation of similarity between the sets of the grandchild components can be completed using statistical techniques, which include but are not limited to Jaccard class similarity.

[0025] In some embodiments of this method, the process of determining whether a particular component is dynamic involves detecting whether subcomponents sharing the same parent node include randomly generated character sequences. As the probability that such subcomponents are randomly generated increases, the probability that such subcomponents are dynamic also increases.

[0026] Various embodiments of the present invention provide a system for generating a set of API endpoints associated with a web application by inspecting network data traffic. The system includes a server, such as a web server or a proxy server, adapted to receive multiple network data requests, the multiple network data requests using the {URL, Method} format to identify command actions supported by the web application. A logic element that can be integrated or separated from the server inspects the http request header of the received network data request to extract the original URL string therefrom. The logic element splits each of the original URL strings into multiple components, and constructs an original component tree, the original component tree consisting of multiple branches formed from nodes, wherein each node is a component of at least one original URL string, the logic analyzes each component in the original component tree to determine whether such component is static or dynamic, the logic modifies the original component tree to replace components determined to be dynamic with a universal identifier, and the logic creates a collapsed component tree by merging the same branches of the original component tree into a single branch. The collapsed component tree provides a desired set of API endpoints.

[0027] In some embodiments of such a system, each original URL string is split into a set of components by detecting a slash character " / " within the original URL string and obtaining a plurality of components from character groups separated by such slash characters.

[0028] In some embodiments of such a system, the set of API endpoints included in the collapsed component tree includes multiple tuples in the form of {CollapsedURL,Method}, where the CollapsedURL represents the URL of the dynamic component replaced by a universal identifier, and Method represents an http method request.

[0029] In various embodiments of such a system, the original component tree includes: at least one parent node, the at least one parent node having a component located closer to the starting point of the original URL string; and at least one child node, the at least one child node having a component located farther from the starting point of the original URL string than the parent node.

[0030] In some embodiments of such a system, the logic element analyzes whether the number of child nodes paired with a particular parent node is relatively large. If the number of child nodes paired with a particular parent node is relatively large, then such child node components are likely to be dynamic.

[0031] Some embodiments of such a system include a memory bank for storing the number of times a particular component appears in a received network data request within a predetermined time period. The logic element analyzes the number of occurrences of a parent node component within the predetermined time period and compares the number of occurrences with the number of occurrences of a paired child node component associated with the parent node within the predetermined time period. If the number of occurrences of a parent node component within the predetermined time period is relatively high, while the number of occurrences of each paired child node associated with the parent node within the predetermined time period is relatively low, then such paired child node components are likely to be dynamic.

[0032] In some embodiments of this system, the logic element detects whether a specific child node component (which shares the same parent node) is dynamic by determining the similarity of the set of corresponding grandchild node components relative to each other. In other words, focus on a specific parent node, find out its child nodes, and then for each such child node, form a set of corresponding grandchild components. For example, if parent node A has 3 child components X, Y and Z, and if each of the child components X, Y and Z has its own child node component (that is, the grandchild node of parent node A), the logic element calculates the similarity between the set of the child nodes of X, the child nodes of Y and the child nodes of Z. If it is found that there is a relatively high similarity between the sets of these grandchild components, the child node components X, Y and Z may be dynamic. The calculation of similarity between the sets of the grandchild components can be completed using statistical techniques, which include but are not limited to Jaccard class similarity.

[0033] In some embodiments of such a system, the logic element detects that child node components sharing the same parent node have randomly generated character sequences. In such cases, such child node components are considered dynamic.

[0034] It will be appreciated that various techniques for detecting whether a particular child node component is static or dynamic can be combined. For example, a weighted average of such techniques can be calculated and the weighted average can be compared with a threshold. If the weighted average equals or exceeds the threshold, the component is considered to be dynamic, and similar URL strings are folded.

[0035] The foregoing and other features and advantages of the invention will become more apparent from the following more particular description of specific embodiments of the invention as illustrated in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] A more complete understanding of the present invention may be obtained by referring to the detailed description and claims when considered in conjunction with the accompanying drawings, in which: Figure 1is a high-level flow diagram illustrating a series of network data requests that use the {URL, Method} format to identify command actions supported by a web application, and illustrates how such URLs may be collapsed after dynamic components are detected. Figure 2 is a diagram showing how a set of URLs can be represented as a tree structure. Figure 3 is a high-level diagram showing a client computer communicating over the Internet with a reverse proxy server, which in turn communicates with three web application host servers. Figure 4 is a flow chart showing the initial steps for processing http header request information obtained from each network data request. Figure 5 is a flow chart illustrating the steps performed to determine whether a particular URL string component is dynamic or non-dynamic according to one embodiment of the present invention. DETAILED DESCRIPTION

[0037] Embodiments described herein involve inspecting so-called "Layer 7" (L7) network data traffic to dynamically generate a set of API endpoints, which are exposed by a web application, using the {URL, Method} format to uniquely identify command actions supported by the web application. The API endpoints are obtained simply by inspecting HTTP request headers of requests known to have been successfully served by the application, without any human intervention or configuration. The methods and systems of the present invention successfully identify dynamic components with high accuracy and produce a more useful learned set of API endpoints for the purposes described above.

[0038] In the example discussed above, the web application hosted at www.reddit.com uses a unique ID in the form of a seemingly random string to identify a unique object. In the URL string listed below: https: / / reddit.com / r / news / comments / 8zo9or The characters "8zo9or" correspond to a unique ID, or a random string. Attributes associated with this random string may appear later in the URL, as exemplified below: https: / / reddit.com / r / news / comments / 8zo9or / top /

[0039] A web-based application may have thousands of such unique IDs embedded in its URL. However, these thousands of unique IDs do not really contribute any useful information because the functions performed at each such endpoint are the same. Relative to the above example, for the purpose of checking API endpoints, the abbreviated URL "reddit.com / r / news / comments / 8zo9or / top / " is a more useful URL string when compared to the original URAL "reddit.com / r / news / comments / 8zo9or / top / ". The random string "8zo9or" is the dynamic part of the URL string. According to the present invention, the API endpoint mapping program can take this into account by replacing this dynamic part of the URL with a predefined universal string (e.g., "DYN" or "*"). The subdirectories of these dynamic parts of the URL (e.g., the component "top" in the above example) may themselves be static and should therefore be mapped to different API endpoints.

[0040] Based on the previous examples listed above, let us assume that the following URL is associated with the reddit.com website: 1)https: / / reddit.com / r / news / comments / 8zo9or / top / 2) https: / / reddit.com / r / news / comments / 8zo9or / controversial 3) https: / / reddit.com / r / news / comments / 8zo9or / new

[0041] An API endpoint mapper in accordance with the principles of the present invention would convert each such URL into a corresponding collapsed URL as shown in the following table:

[0042] A URL can be decomposed into multiple components by splitting the original URL string using the slash character " / ". The resulting array of strings is referred to herein as a "component". For example, the URL "https: / / reddit.com / r / news / comments / 8zo9or / top / " can be split into the following components: "r"; "news", "comments"; "8zo9or"; and "top". In this example, the component "8zo9or" is dynamic, while the other components are static. The present invention provides a highly accurate method for detecting a set of API endpoints for a website that uses URLs with dynamic URL components.

[0043] The system and method of the present invention uses the {URL, Method} protocol to receive the original URL string including the L7HTTP request header as input, and provides a list of API endpoints as output, each API endpoint being defined as a tuple {CollapsedURL, Method}. In this regard, the term "CollapsedURL" refers to a URL in which a dynamic component marker (such as the string "8zo9or" in the above example) is replaced by a universal placeholder (such as "DYN").

[0044] Figure 1 is a flow chart that generally illustrates input information in the form of raw {URL, Method} "tuples" of multiple requests processed by a server, and corresponding output information in the form of a list of unique {CollapsedURL, Method} tuples. The input request shown in box 100 includes the following: POST http: / / waIdemo.com / apt / login / '? user=abcd&pwd=secret GET http: / / wafdemo.com / api / user / abcd / GET http: / / wal;demo.com / api / user / abcd / logout POST http: / / wafdemo.com / api / login / ? user=-abcd&pwd=secret GET http: / / wafdemo.com / api / user / abcd / vehicle / 1234 GET http: / / wafdemo.com / api / user / abcd / vehicle / 5678 GET http: / / wafdemo.com / api / user / abcd / vehicle / images / profile.jpg

[0045] exist Figure 1 In FIG. 1 , block 102 (“URL Folder”) conceptually represents the work performed in accordance with the methods and systems of the present invention. Block 104 shows the result of the folded URL request after merging the dynamic components of the original URL string into a set of five folded endpoints. Figure 1In box 104, line 106 covers the two original POST requests. Line 108 covers the GET request http: / / wafdemo.com / api / user / abcd / . Line 110 covers the GET request and http: / / wafdemo.com / api / user / abcd / logout. Line 112 covers the two GET requests ending with the dynamic strings "1234" and "5678". Finally, line 114 covers the final GET request http: / / wafdemo.com / api / user / abcd / vehicle / images / profile.jpg.

[0046] To help explain the way in which dynamic components are identified in each URL string, it is helpful to first understand the concepts of component tree structure and dynamic component properties. These concepts will now be described. Component tree structure

[0047] A URL set can be represented as a tree structure consisting of nodes, where each node is a component in at least one URL in the set. Figure 2 An example of such a tree structure is shown in Figure 2 , the leftmost column 200 represents the parent node "api", which is the node closest to the "root" of the tree, and in this case, is common to all URL strings represented by the tree. Figure 2 Column 208 on the right side of represents the "leaves" of the tree farthest from the root of the tree. Column 202 includes three "child" nodes relative to parent node 200. The first child node "user" spans three rows 210, 212 and 214. The second child node "vehicle" is listed in row 216. The third child node, also referred to as "vehicle", is listed in row 218. Column 204 can be considered a "grandchild" node relative to parent node column 200. However, it will be understood that the term "parent node" can be applied to entries in column 202, for example, in which case the entries of column 204 can be considered "child" nodes relative to the components shown in column 204, and the entries of column 206 can be considered "grandchild" nodes thereof.

[0048] Figure 2 The leaves in the tree shown (ie, the components in column 208) map to the following set of URLs: 1. / api / user / f5om89s / vehicle / sdvkj1 2. / api / user / f5om89s / comment / ncu123 3. / api / user / f5om89s / profile 4. / api / vehicle / asda123 / picture / abcd1234 5. / api / vehicle / f50m89s / registration / 597usj Dynamic component properties

[0049] When a URL set is represented as a tree structure, such as relative to Figure 2 As discussed, the identification of dynamic components is facilitated by the following set of rules: 1. The term "NumChildren" will be used in this article to refer to the number of child nodes paired with the corresponding parent node. If the NumChildren of a parent node is relatively large, the child hierarchy of this parent node is likely to be dynamic. 2. The term "NumOccurrences" will be used in this article to refer to the number of times a particular component appears in a request within a given period (e.g., within a 24-hour period). If a subcomponent and its sibling components have a low NumOccurrences, but their respective parent components have a high NumOccurrences, then the subcomponent and its sibling components are more likely to be dynamic. 3. In the world of statistical analysis, there is a tool called the Jaccard similarity index (sometimes called the "Jaccard similarity coefficient"), which compares the members of two sets of data to see which members are common between the two groups and which members are different. Therefore, it is essentially a measure of the similarity of two sets of data, ranging from 0% to 100%; the higher the percentage, the more similar the two groups will be evaluated. If two grandchild node components (both sharing the same parent node) have a high Jaccard similarity index, the intermediate child node component is more likely to be dynamic. Additional details about the use of the Jaccard similarity index are provided below. Although Jaccard similarity is conventionally applied to two sets of data, the same concept can be applied to measure the similarity between three or more sets of data. As used herein, the term "Jaccard class similarity" should be understood to refer to the application of conventional Jaccard similarity analysis applied to three or more sets of data. 4. Finally, if the various character sequences of a component and all its sibling components appear to be randomly generated, then the component and its sibling components may be dynamic. The following will describe in more detail how to detect such random generation.

[0050] Now go to Figure 3, shows a high-level diagram of a typical proxy server deployment. Client computer 300 is interconnected to proxy server 304 via Internet 302. Proxy server 304 is in turn interconnected with three web application servers 306, 308 and 310. Proxy server 304 acts as a front end for web applications and can optionally provide additional functions such as load balancing, firewall protection, Bot detection, etc. Proxy server 304 receives all network data requests for web application servers 306, 308 and 310, and is able to inspect and extract the http headers of all requests flowing through proxy server 304. The method of the present invention can be executed within proxy server 304 or within a computer processor to which it is coupled. Determining the API endpoint within proxy server 304 can provide many of the benefits mentioned above, including data profiling and user behavior profiling, as well as being used as a tool that can be used by system administrators and security and site reliability engineering teams. As mentioned, API endpoint information is valuable for security purposes, such as: 1) "whitelisting" of known API endpoints; 2) profiling of known API endpoints; 3) user behavior analysis using API endpoint access sequences; and 4) detection of attempted denial of service (DoS) attacks. In addition to security considerations, such API endpoint information is also valuable for the following purposes: 1) identifying the slowest API endpoints; 2) identifying those most prone to errors; and 3) capacity planning.

[0051] exist Figure 4 The initial steps for processing the http header request information are shown in the flowchart of FIG. At step 400, the next network data request is received by the proxy server 304. At step 402, the URL string and Method are extracted from the http header information. For example, if the header of the request includes a "tuple" {URL, Method} in the form of GET http: / / wafdemo.com / api / user / abcd / , the URL string wafdemo.com / api / user / abcd / and the Method "GET" are extracted.

[0052] exist Figure 4 At step 404, the extracted URL string is "normalized" to remove any extra symbols and resolve any path traversals. At step 406, the normalized URL string is split into its components by searching for slash characters, which separate such components; in the above example, the components are "api", "user", and "abcd". At step 408, a component tree table storing such URL strings is determined (see, for example, Figure 2) already contains components arranged to form the current URL string; if not, the current component combination is added to the component tree table and control is transferred to step 412. If the current component combination is already included in the component tree table, control is transferred directly to step 412. At step 412, the "Method" (e.g., "GET") extracted from the current request header is saved in the form of metadata associated with the last or "leaf" component node of a particular branch of the component tree to remember all Methods observed for a given URL string; in this regard, it will be recalled that the API endpoint identification includes a unique combination of URL and Method.

[0053] Still reference Figure 4 , control passes from step 412 to decision step 414 to determine whether it is time to re-cluster the URL strings represented in the tree structure table. For example, one might decide to re-cluster the entries in the tree component table for every ten (or optionally, every hundred) insertions of new entries in the table. If the decision is "no," control returns to step 400 and a new network data request is received. If the decision is "yes," the URL re-clustering process is performed in the manner described below. Figure 4 The logic of the steps described in can be considered a “clustering module”.

[0054] After the specified number of insertions into the new component table has been reached, the URL re-clustering process is performed. Figure 5 The process is described by the flowchart of . In general, the re-clustering process "traverses" the nodes of the URL component tree table, and at each node with child nodes, uses Figure 5 The algorithm outlined in determines whether a node's children are dynamic.

[0055] Reference now Figure 5 , the process begins at step 500: checking the first parent node in the tree structure table. At step 502, the number of child component nodes (NumChildren) is calculated for the selected parent node. If NumChildren is greater than the specified number of child nodes (maximum limit), control is transferred to box 518, and the child node is determined to be dynamic. This is because the parent node has an abnormally high number of child nodes indicating that the child level of this parent node is dynamic.

[0056] If step 502 determines that NumChildren does not exceed the maximum limit, control passes to step 504 to calculate NumChildrenDynScore. Dynamic components are often much larger than the number of non-dynamic components. In order to find a parent node whose number of child components is much larger than other nodes, a method based on the so-called Z-score can be used. A "Z-score" (also known as a "standard score") indicates how many standard deviations an element differs from the mean. The z-score can be calculated as z = (X-μ) / σ, where z is the z-score, X is the value of the element, μ is the population mean, and σ is the standard deviation. For more information, see "Statistics HowTo" at the following location: https: / / www.statisticshowto.datasciencecentral.com / probability-and-statistics / z-score / .

[0057] When calculating the Z-score, NumChildMean and NumChildSTD (standard deviation) are calculated for each parent node in the tree. When visiting a node, its Z-score is used to compare the node's children to other nodes in the tree. The higher the Z-score, the more likely the node is dynamic. After calculating the mean value NumChildMean and the standard deviation value NumChildSTD, the Z-score can be calculated according to the following formula: Z-score=(NumChild-NumChildMean) / NumChildSTD. This Z-score is then used as NumChildDynScore in step 504 .

[0058] Control then passes to step 506 to calculate the EntropyDynScore. The EntropyDynScore considers the number of occurrences of a parent node component versus an associated child node component within a predetermined time period (e.g., in a set of access logs). Figure 1 In the earlier example shown, two similar GET requests are listed, namely, GET http: / / wafdemo.com / api / user / abcd / vehicle / 1234 GET http: / / wafdemo.com / api / user / abcd / vehicle / 5678

[0059] If the parent node being analyzed is a static component "vehicle", and the dynamic child node components are "1234" and "5678", the number of occurrences per unit time of the component "vehicle" will usually be much greater than "1234" or "5678". In addition, in the case where the child components are dynamic, the occurrences per unit time of such dynamic components are more likely to be more evenly distributed (compared to each other) than non-dynamic components. The degree to which the occurrences are evenly distributed among such child components can be measured by measuring the entropy of the distribution of occurrences among such child components. The more evenly distributed the occurrences between child nodes, the more likely such child components are dynamic. The calculation of EntropyDynScore first requires the calculation of entropy, as well as the mean and standard deviation values ​​of entropy. Entropy can be calculated according to the following formula:

[0060] where H(X) is a measure of the entropy of an event X with n possible outcomes and probabilities p_1,…,p_n. Additional details are provided in Sebastian Kwiatkowski, “Entropyi is a measure of uncertainty”, Toward Data Science, October 6, 2018, available at: https: / / towardsdatascience.com / entropy-is-a-measure-of-uncertainty-e2c000301c2c.

[0061] After calculating the entropy of each child node and calculating the mean EntropyMean and standard deviation EntropyStd, the EntropyDyn score can be calculated according to the following formula: EntropyDynScore=[(NumAccess / NumChild)-EntropyMean] / EntropyStd Where NumAccess is the number of times a URL with a particular parent node component is accessed (ie, the number of times this parent node component appears in network traffic within a predetermined time period); and NumChild is the number of different child node components associated with this particular parent node.

[0062] Still reference Figure 5, control then goes from step 506 to step 5084 to calculate StringClassifierDynScore. Typically, the dynamic component of the URL path string is a string of randomly generated characters and / or integers. A string classifier based on the so-called RNN LSTM can be used to classify the text of the child node component to predict whether the child node component is a dynamic component. RNN LSTM is a technology based on deep neural networks that is used to train classifiers and can then classify strings after training with a larger corpus of labeled training samples. In this case, two types of training data sets labeled "dynamic" or "non-dynamic" are used to train the classifier. The trained classifier then processes the strings that constitute the child component (and all its sibling components) to determine whether they are dynamic or non-dynamic. Additional details about the RNN LSTM classifier are provided in the following: "Understanding LSTM Networks", Colah's Blog, published on August 27, 2015: https: / / colah.github.io / posts / 2015-08-Understanding-LSTMs / and “The Unreasonable Effectiveness of Recurrent Neural Networks”, AndrejKarpathyBlog, published on May 21, 2015: http: / / karpathy.github.io / 2015 / 05 / 21 / rnn-effectiveness /

[0063] This type of string classifier renders predictions by providing a score between the values ​​0 and 1, where "0" is non-dynamic and "1" is dynamic. The string classifier is run on all child node strings. Then, the value NumDynChildNodes is calculated, where NumDynChildNodes is equal to the total number of child nodes where the string classifier prediction score is greater than 0.95. Next, the average is calculated to get the average score of all child nodes of a particular parent node, as listed below: StringClassifiedDynScore=NumDynChildNodes / NumChild Where NumChild is the number of child nodes associated with the parent node under study. The average calculated above is used as StringClassifiedDynScore.

[0064] Then, control is transferred from step 508 to decision diamond box 510 to determine whether, for a given parent node under study, its child nodes have their own child nodes, that is, whether the parent node under study has a grandchild node. If there is no grandchild component, step 512 is bypassed. If there is a grandchild component, control is transferred to step 512 to calculate the JaccardSimilarityDynScore. The theoretical basis of step 512 is that, when a group of child nodes is dynamic, the grandchild components of the group of child nodes are usually very similar to each other. Therefore, for each child node associated with the parent node under study, a group of grandchild component strings is compiled. The Jaccard class similarity index is then calculated to compare the similarity (or dissimilarity) of the grandchild component string groups associated with such child nodes; a relatively high Jaccard class similarity index indicates that the intermediate child node located between the parent node and the grandchild node is dynamic.

[0065] When comparing two sets of data, the Jaccard similarity index can be expressed as the formula listed below: Jaccard similarity index = [NB / TM] x 100, Where NB is the number of data members that appear in the two data groups being compared; Where TM is the total number of different data members that appear in either or both of such data groups being compared; and Where the result is expressed as a percentage. This resulting percentage indicates how similar (or different) the two data groups are relative to each other. For example, if two data groups each containing 100 members are identical to each other, then NB=100, TM=100, and the result=100%. As another example, if two data groups each containing 100 members have no common members, then NB=0, TM=200, and the result=0%. As a final example, if two data groups each contain 100 members, and 50 members appear in both data groups, then NB=50, TM=150 (i.e., there are 50 unique members in the first data group, 50 unique members in the second data group, and the two data groups share 50 common members), and the result is 33.3. Once again, the higher the resulting percentage, the more similar the two data groups are to each other. Although this example uses two groups of data, the same concept can be applied to and can be extended to three or more groups of data, and this application is referred to as Jaccard-like similarity in this article. Although Jaccard similarity is a type of statistical tool that can be used to compare similarities and differences between multiple groups of data, other comparison tools known to those skilled in the art can also be used.

[0066] Still reference Figure 5In step 512, the JaccardSimilarityDynScore covering all grandchild components of each parent node is calculated according to the following formula: (intersection of all grandchild components) / (union of all grandchild components).

[0067] The result of the above expression is used as the JaccardSimilarityDynScore of the child nodes of the parent node under study. The higher the JaccardSimilarityDynScore of a particular parent node under study, the more likely the child node components are dynamic.

[0068] Then, control passes to step 514, where a weighted average, called "DynScore", is calculated and compared to a threshold to determine that the node's children are dynamic components and, therefore, such children should be collapsed using a subtree merge operation. DynScore can be expressed as follows: DynScore=W1 x NumChildDynScore+W2 x EntropyDynScore +W3 x StringClassifierDynScore +W4 x JaccardSimilarityDynScore, Where W1, W2, W3 and W4 are weight factors used to scale the scores to make them comparable to each other. At step 516, the total DynScore is compared to a threshold, and if the DynScore exceeds the threshold, the group of subcomponents under evaluation is considered dynamic; if not, the group of subcomponents under evaluation is considered not dynamic. Control returns to step 500, and the next node in the tree structure table is evaluated in the same manner.

[0069] exist Figure 5 In the embodiment described in , the DynScore is influenced by each of the four component scores described above, and in combination, such component scores collectively provide a highly reliable indication of whether a node is dynamic or non-dynamic. However, those skilled in the art will appreciate that one or more of such component scores may be omitted while continuing to accurately predict whether a component node is dynamic or non-dynamic.

[0070] If the child nodes of the parent node are determined to be dynamic, a node collapse operation is performed within the node tree structure table so that all subtrees of the child nodes of this node are merged into a single subtree, and the dynamic nodes of the single subtree are assigned a universal identifier, such as "DYN". This has the effect of replacing multiple child node subtrees with a single dynamic component child node subtree. After the tree collapse operation is completed, the path to each leaf node represents a collapsed URL. At this leaf node, more metadata related to this collapsed URL is stored, including the HTTP Method combination associated with the collapsed URL string. The resulting component tree table provides the learned set of API endpoints, as identified by the resulting {CollapsedURL,Method} combination.

[0071] The resulting set of API endpoints provides a wealth of information that can be used to profile data usage, user behavior, provide security information, help prevent malicious threats, detect errors associated with web applications, and assist in capacity planning.

[0072] The computing systems mentioned herein (including but not limited to proxy servers) may include integrated circuits, microprocessors, personal computers, servers, distributed computing systems, communication devices, network devices, firewalls, proxy servers, web servers, application gateways, stateful connection managers, and / or various combinations thereof. For example, the processors mentioned herein may include microprocessors. The chipsets mentioned herein may include one or more integrated circuits, and the memories and storage bodies mentioned herein may include volatile and / or non-volatile memories, such as random access memories (RAMs), dynamic random access memories (DRAMs), static random access memories (SRAMs), magnetic media, optical media, nano media, hard disk drives, compact disks, digital versatile disks (DVDs), and / or other devices configured to store analog or digital information, such as stored in a database. Therefore, it will be appreciated that various instances of the above logic may include hardware, firmware or software stored on a computer-readable medium, or a combination thereof.

[0073] As used herein, the term "logic" or "logic element" may refer to a server or a separate computing system coupled to a server. Such logic may include a computer processor, associated storage, and associated input ports and output ports. Various instances of the logic described herein may include hardware, firmware or software stored on a computer-readable medium, or a combination thereof. Such logic may be implemented in an electronic device to produce a dedicated computing system. The computer-implemented steps of the methods described herein may include a set of instructions stored on a computer-readable medium that, when executed, causes a computing system to perform the steps. As used herein, computer-readable media refers only to non-transitory media, does not cover signal transmissions in transitory forms, and expressly excludes paper.

[0074] A computing system that is programmed to perform specific functions according to instructions from program software is a special-purpose computing system for performing those specific functions. The data manipulated by the special-purpose computing system in performing those specific functions is stored at least electronically in a cache of the computing system, with the special-purpose computing system physically changing from one state to the next with each change in the stored data. Claims directed to methods herein are expressly limited to computer-implemented embodiments thereof, and expressly do not cover embodiments that may be performed purely mentally.

[0075] The absence of the term "means" in any claim should be construed to exclude the claim from being interpreted under Section 112(f) of the Patent Act. As used in the claims of this application, "configured to" and "configured for" are not intended to invoke Section 112(f) of the Patent Act.

[0076] Several embodiments are specifically described and / or described herein to illustrate specific applications of the present invention. These descriptions and drawings should not be considered to have a restrictive meaning, as it should be understood that the present invention is by no means limited to the disclosed embodiments. It will be appreciated that various modifications or adjustments of the methods and / or specific structures described herein may become apparent to those skilled in the art. All such modifications, adjustments or variations are considered to be within the spirit and scope of the present invention and within the scope of the appended claims.

Claims

1. A method for generating a set of API endpoints by inspecting network data traffic, the method comprising the following steps: Receiving a plurality of network data requests, wherein the plurality of network data requests use a {URL, Method} format to identify command actions supported by a web application; Checking HTTP request headers of the received plurality of network data requests to extract original URL strings therefrom; Splitting each of the original URL strings into a plurality of parts; Constructing an original component tree consisting of a plurality of branches formed from nodes, wherein each node is a component of at least one of the original URL strings; Using statistical calculations of components, based on node similarity, to determine whether nodes of multiple branches are dynamic or static; clustering the original component tree into a collapsed component tree based on the determination; and A set of API endpoints are derived from the collapsed component tree.

2. The method according to claim 1, wherein: The multiple network data requests are received in real time from data packets transmitted over the network.

3. The method according to claim 1, wherein: The plurality of network data requests are received as an offline access log generated by a network server.

4. The method according to claim 1, wherein: The step of splitting each of the original URL strings into a plurality of components comprises: detecting a forward slash character " / " in the original URL string, and obtaining the plurality of components from character groups separated by such forward slash characters.

5. The method according to claim 1, wherein: The obtained API endpoint set includes multiple tuples in the form of {CollapsedURL, Method}, wherein the Collapsed URL represents a URL in which a dynamic component is replaced by a universal identifier, and Method represents an http method request.

6. The method according to claim 1, wherein: The original component tree includes at least one parent node, the at least one parent node includes a component located closer to the starting point of the original URL string, wherein the original component tree includes at least one child node, the at least one child node includes a component located farther from the starting point of the original URL string than the parent node.

7. The method according to claim 6, wherein: The step of analyzing each component in the original component tree to determine whether such component is dynamic includes determining whether the number of child nodes paired with a parent node is large.

8. The method according to claim 7, further comprising the steps of: Tracking the number of times a particular component appears in received network data requests within a predetermined time period, wherein the step of analyzing each component in the original component tree to determine whether such component is dynamic includes detecting that a component of a parent node has a high number of occurrences within the predetermined time period, while a corresponding child node associated with the parent node has a low number of occurrences.

9. The method according to claim 6, wherein: The step of analyzing each component in the original component tree to determine whether such component is dynamic includes: detecting whether there is a high similarity between components of grandchild nodes sharing the same parent node, and determining that the intermediate child node component is dynamic when such a high similarity is found.

10. The method according to claim 1, wherein: The step of analyzing each component in the original component tree to determine whether such component is dynamic includes: detecting whether child components sharing the same parent node include a randomly generated character sequence.

11. A system for generating a set of API endpoints by inspecting network data traffic, the system comprising the combination of: A server, the server receiving a plurality of network data requests, the plurality of network data requests using a {URL, Method} format to identify command actions supported by a web application; and Logic, the logic inspecting the HTTP request headers of the plurality of network data requests received to extract original URL strings therefrom, the logic splitting each of the original URL strings into a plurality of components and constructing an original component tree consisting of a plurality of branches formed by nodes, wherein Each node is a component of at least one original URL string, the logic analyzes each component in the original component tree using statistical calculations of components based on similarity of nodes to determine whether the component is static or dynamic, and the logic clusters the original component tree into a collapsed component tree, The collapsed component tree provides a set of API endpoints.

12. The system according to claim 11, wherein: The logic is contained in the server.

13. The system according to claim 11, wherein: The logic separates each of the original URL strings into multiple components by detecting forward slash characters " / " in the original URL strings and obtaining multiple components from character groupings separated by such forward slash characters.

14. The system according to claim 11, wherein: The API endpoint set included in the collapsed component tree includes multiple tuples in the form of {CollapsedURL, Method}, wherein the CollapsedURL represents the URL of the dynamic component replaced by a universal identifier, and Method represents an http method request.

15. The system according to claim 11, wherein: The original component tree includes at least one parent node, the at least one parent node includes a component located closer to the starting point of the original URL string, wherein the original component tree includes at least one child node, the at least one child node includes a component located farther from the starting point of the original URL string than the parent node.

16. The system of claim 15, wherein: When determining whether a component in the original component tree is dynamic, the logic analyzes whether the number of child nodes paired with the parent node is large.

17. The system according to claim 16, comprising: A memory for storing the number of times a particular component appears in received network data requests within a predetermined time period, wherein the logic analyzes the number of times a component of a parent node appears within the predetermined time period, and analyzes the number of times a corresponding child node associated with the parent node appears within the predetermined time period, and if the number of times the component of the parent node appears within the predetermined time period is high and the number of times the corresponding child node associated with the parent node appears within the predetermined time period is low, then the logic determines that the child node component is dynamic.

18. The system of claim 17, wherein: The logic detects whether there is a high similarity among components of grandchild nodes that share a common parent node, and determines that the intermediate child node component is dynamic when such a high similarity is found.

19. The system of claim 18, wherein: The logic detects subcomponents that share the same parent node and also include randomly generated character sequences to determine that such subcomponents are dynamic.

Images