API asset identification method and storage medium

Abstract

The invention relates to an API asset identification method and a storage medium, and the API asset identification method comprises the steps: obtaining a URL path set in network traffic; performing first grouping based on the domain name and the path depth to obtain a first URL group, determining a grouping basis based on the deduplication number of the path nodes at each level, and performing second grouping to obtain a second URL group; path node feature analysis and clustering analysis are carried out on each second URL group, and universal path nodes are identified; and generating an API asset list based on the universal path nodes. Through the method and the device, the problems of inaccurate API asset identification, high calculation complexity and low efficiency caused by the fact that a Restful style API has a large number of general configuration paths in the prior art are solved, the API assets of the general configuration paths are automatically, accurately and efficiently extracted from massive network traffic, and the automation level and the accuracy of enterprise API asset management are effectively improved.

Description

Technical Field

[0001] This application relates to the field of cybersecurity, and in particular to API asset identification methods and storage media. Background Technology

[0002] API (Application Programming Interface) assets refer to the sum of API resources owned and managed by an enterprise or organization in its digital business operations. These include the API's code implementation, documentation, access permissions, data interaction rules, and related security configurations, forming a crucial component of an enterprise's digital capabilities. Currently, API asset management is a major challenge in the field of cybersecurity. Enterprises utilize numerous internal and external systems involving a large number of API assets, and the rapid iteration of internet applications also results in a significant amount of incremental and invalid API assets. These exposed API assets often become the entry point for cyberattacks against enterprise information systems, and the security of API assets directly impacts the security of enterprise information systems and data assets.

[0003] The existing methods of API asset management generally include the following: (1) relevant departments collect data from various system teams. This method relies on manual management, and the timeliness and completeness of the data are difficult to guarantee; (2) management is achieved by adding API asset management plugins to the code. This method is more effective for a single system. In actual production, there are a large number of systems with different technology stacks. Various systems follow different specifications, which makes implementation difficult and depends on the developers' compliance with the specifications; (3) potential API assets are identified based on code analysis. This method is technically difficult to implement and has similar problems to the former; (4) API assets are identified based on real traffic analysis of API asset requests. This method has advantages such as high timeliness and real and valid assets. However, since a large number of Restful style APIs now have wildcard paths, how to identify wildcard paths in the URL to accurately identify the API assets themselves has become a major problem.

[0004] Specifically, in the aforementioned technical approach based on real traffic, traditional methods typically employ URL path analysis based on tree structures (such as trie or prefix tree), handling hierarchical relationships by constructing a tree structure of URL paths. However, this method requires building and maintaining complex tree data structures, resulting in high memory consumption and low computational efficiency when processing massive amounts of URLs (Uniform Resource Locators). The time complexity increases rapidly with the number of URLs and data dimensions, leading to high computational complexity. Furthermore, the complex algorithms for traversing tree structures and splitting / merging nodes make code implementation and maintenance difficult, requiring handling complex boundary conditions and exceptions, thus making its implementation quite challenging.

[0005] Currently, no effective solution has been proposed to address the issues of high computational complexity and implementation difficulty in identifying wildcard URL paths in related technologies. Summary of the Invention

[0006] This application provides an API asset identification method and storage medium to at least solve the problems of high computational complexity and high implementation difficulty when identifying API assets based on URL paths in related technologies.

[0007] In a first aspect, embodiments of this application provide an API asset identification method, including:

[0008] Retrieve the set of URL paths from network traffic;

[0009] Based on the domain name and path depth of the URL path, the set of URL paths is first grouped to obtain multiple first URL groups;

[0010] For each first URL group, the grouping criteria are determined based on the number of deduplications of the path nodes in the current first URL group at each path depth level in the corresponding URL path, and a second grouping is performed based on the grouping criteria to obtain multiple second URL groups.

[0011] Cluster analysis is performed on the path nodes within each second URL group to identify wildcard path nodes;

[0012] Based on the wildcard path nodes, a wildcard path API asset list is generated.

[0013] In some embodiments, determining the grouping criteria for each first URL group based on the number of duplicates of path nodes in the current first URL group at each path depth level of the corresponding URL path includes:

[0014] Within the current first URL group, each URL path is parsed into a first URL path array consisting of path node values. A first path node matrix is ​​constructed with the first URL path array as the row and the path depth level of the current first URL group as the column.

[0015] Calculate the number of duplicates for each column of path nodes in the first path node matrix to obtain a first deduplication count array. Each element in the first deduplication count array corresponds to the number of duplicates for each column of the first path node matrix.

[0016] Calculate the baseline value based on the first deduplication count array;

[0017] By sequentially traversing the first deduplication count array, a continuous sequence of elements starting from the first element of the first deduplication count array is identified, wherein the value of each element in the continuous sequence of elements is not greater than the benchmark value;

[0018] The columns of the first path node matrix corresponding to the continuous element sequence are determined as grouping key columns;

[0019] The path level corresponding to the grouping key column is determined as the grouping basis.

[0020] In some embodiments, calculating the baseline value based on the first deduplication count array includes:

[0021] Calculate the mean and median of all elements in the first deduplication array to obtain the first mean and the first median;

[0022] The minimum value between the first average and the first median is determined as the benchmark value.

[0023] In some embodiments, the step of clustering path nodes within each second URL group to identify wildcard path nodes includes:

[0024] Within the current second URL group, each URL path is parsed into a second URL path array consisting of path node values. A second path node matrix is ​​constructed with the second URL path array as the rows and the path depth level of the current second URL group as the columns.

[0025] Calculate the number of duplicates for each column of path nodes in the second path node matrix to obtain a second deduplication count array. Each element in the second deduplication count array corresponds to the number of duplicates for each column of the second path node matrix.

[0026] Based on the preset dimensionality reduction rules, the dimensionality reduction process is performed on each path node value in the second path node matrix to obtain the dimensionality reduction array corresponding to each path node value;

[0027] For each column in the second path node matrix, calculate the statistical characteristics of the current column. The statistical characteristics include the second number of duplicates and the second average value of the current column. The second average value is the numerical average value calculated based on the dimensionality reduction array of all path node values ​​of the current column.

[0028] Based on the second deduplication count array and the second average value of each column, the target columns that need to be clustered are marked from all columns of the second path node matrix;

[0029] Clustering analysis is performed on the target column using a clustering algorithm to obtain the clustering analysis results;

[0030] Based on the clustering analysis results, path nodes within the same cluster are identified as wildcard path nodes under the same wildcard pattern.

[0031] In some embodiments, the dimensionality reduction processing of each path node value in the second path node matrix based on a preset dimensionality reduction rule includes:

[0032] Type identification is performed on the current path node values ​​in the second path node matrix;

[0033] If the current path node value is identified as a predefined special type, then the current path node value is mapped to a preset fixed numerical array. The special type includes at least one of pure numbers, UUIDs, hexadecimal strings, and file resource names.

[0034] If it is not identified as the special type, then according to the predefined character mapping table, each character in the string of the current path node value is converted into the corresponding numerical value, and the current path node value is formed into a reduced-dimensional array in order;

[0035] The character mapping table defines different numerical mapping relationships for different character sets.

[0036] In some embodiments, the step of identifying the target columns for clustering analysis from all columns of the second path node matrix based on the second deduplication count array and the second average value of each column includes:

[0037] The current column in the second path node matrix is ​​identified as the target column and marked when it meets at least one of the following rules:

[0038] Rule 1: If the path depth corresponding to the current second URL group is 1;

[0039] Rule 2: If the second average value of the current column is less than the preset clustering threshold;

[0040] Rule 3: The second deduplication count in the current column is greater than the median of the second deduplication count array, and is greater than or equal to the sum of the average value and the adjustment value of the second deduplication count array;

[0041] The adjustment value is calculated as follows: the standard deviation of the second deduplication array is multiplied by a preset coefficient, and then multiplied by the serial number of the second deduplication value in the current column in the second deduplication array in ascending order of numerical value.

[0042] In some embodiments, applying a clustering algorithm to the target column to perform clustering analysis and obtain clustering results includes:

[0043] Get the number of path nodes and average character length of the current target column;

[0044] Based on the number of path nodes and the average character length, the neighborhood radius parameter eps and the minimum number of points parameter min_pts of the DBScan algorithm are determined through predefined calculation rules.

[0045] The DBScan algorithm, configured with the parameters eps and min_pts, is used to perform cluster analysis on the target column to obtain the cluster analysis results.

[0046] In some embodiments, the application is configured with the DBScan algorithm, using the eps and min_pts parameters, to perform cluster analysis on the target column, and further includes:

[0047] During the DBScan algorithm's cluster expansion process, when the number of points in the neighborhood of the current core point meets the target condition for the first time, a portion of points are extracted from the neighborhood point set of the current core point according to a preset sampling rule to form a sampling point set, and subsequent neighborhood queries and cluster expansion operations are only performed on the sampling point set.

[0048] The target conditions are as follows: the number of points in the neighborhood of the core point is greater than a preset absolute number threshold, and the proportion of the number of points in the neighborhood to the total number of points in the current target column is greater than a preset density region proportion threshold.

[0049] In some embodiments, after obtaining the clustering analysis results, the method further includes:

[0050] Based on the number of members in each cluster in the clustering analysis results, and combined with a dynamic threshold, clusters with a number of members not less than the dynamic threshold are selected to obtain the first set of verification results.

[0051] Based on the first set of verification results, and combined with the preset API keyword library, clusters in which the proportion of keywords in the API keyword library contained in the path node values ​​does not exceed a preset keyword proportion threshold are selected to obtain the second set of verification results.

[0052] The step of identifying path nodes within the same cluster as wildcard path nodes under the same wildcard pattern based on the clustering analysis results includes:

[0053] Based on the second set of verification results, all path nodes within the clusters in the second set of verification results are identified as wildcard path nodes under the same wildcard pattern.

[0054] The dynamic threshold is determined as follows: the total number of URLs in the current second URL group and the column number of the current target column in the second path node matrix are obtained; the ratio of the total number of URLs to an adjustment denominator is calculated as the dynamic threshold; wherein the adjustment denominator is the maximum value of 2 and the square of the column number.

[0055] Secondly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the API asset identification method as described in the first aspect above.

[0056] Compared to related technologies, the API asset identification method provided in this application intelligently segments massive URL paths into structurally similar groups through two groupings, and then performs accurate clustering analysis within each group. This solves the problems of computational complexity, implementation difficulty, and poor wildcard path identification effect of traditional methods, achieving the technical effects of significantly reducing computational complexity, greatly reducing implementation difficulty, and effectively improving the accuracy of wildcard path identification.

[0057] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0059] Figure 1 This is a block diagram of the terminal hardware structure of the API asset identification method according to an embodiment of this application;

[0060] Figure 2 This is a flowchart of an API asset identification method according to an embodiment of this application;

[0061] Figure 3 This is a structural block diagram of an API asset identification device according to an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0063] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0064] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0065] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for an API asset identification method according to an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0066] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the API asset identification method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0067] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0068] This embodiment provides an API asset identification method. Figure 2 This is a flowchart of the API asset identification method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0069] Step S201: Obtain the set of URL paths in the network traffic.

[0070] Network traffic refers to the actual HTTP / HTTPS request data streams generated by an enterprise information system in a real production environment.

[0071] Specifically, complete URLs are extracted from network traffic mirroring, gateway logs, or proxy logs to obtain URL data from network traffic. Each complete URL is parsed to extract its domain name and complete path portion. The path depth is calculated based on this complete path portion. Simultaneously, using string parsing techniques, the protocol (https: / / ), domain name and port (www.example.com:8080), query parameters (?id=123), and fragment identifier (#section) are removed from the complete URL (e.g., https: / / www.example.com:8080 / api / v1 / users?id=123#section) to obtain the clean URL path ( / api / v1 / users). This forms a set of data units, each including the domain name, path depth, and clean URL path. The clean URL paths of all data units constitute a URL path set, where each clean URL path is associated with its domain name and path depth information.

[0072] Step S202: Based on the domain name and path depth of the URL path, the URL path set is grouped for the first time to obtain multiple first URL groups.

[0073] Different domain names typically represent different application systems or service modules, while different path depths reflect the nesting level of URLs within the website structure. Therefore, by first grouping based on these two most macroscopic characteristics, the vast and chaotic collection of URLs can be initially divided into structurally organized groups.

[0074] Specifically, based on the aforementioned domain names and path depth (the number of " / " separators in the path), the above set of URL paths (i.e., the set of clean URL paths) is first grouped to obtain multiple first URL groups. All clean URL paths within each first URL group have the same source domain name and completely consistent path depth, ensuring that the URLs within the group have the same structural framework, laying the foundation for subsequent detailed analysis under a unified dimension.

[0075] Step S203: For each first URL group, determine the grouping criteria based on the number of deduplications of the path nodes in the current first URL group at each path depth level in the corresponding URL path, and perform a second grouping based on the grouping criteria to obtain multiple second URL groups.

[0076] The deduplication count refers to the number of distinct path node values ​​at a specific path depth level. In this scheme, the path level index starts from 0. For example, in the following first URL group, there are two URL addresses, " / api / v1 / users / 123" and " / api / v1 / users / 456". After parsing them into path node arrays, they are ["api", "v1", "users", "123"] and ["api", "v1", "users", "456"], respectively. Then, the deduplication count of path node values ​​at level 3 (i.e., the level with index 3) is 2 ("123" and "456"), while the deduplication count of node values ​​at levels 0 and 1 is 1 ("api" and "v1", respectively). The number of deduplications directly reflects the degree of variation of path nodes at that level. A low number of deduplications indicates that the level mostly consists of fixed values ​​(such as api, v1); a high number of deduplications strongly suggests that the level may contain wildcard parameters (such as various IDs or names).

[0077] Understandably, in a typical URL path structure, the initial path levels often have a low deduplication rate. These levels typically represent relatively stable path structure features, such as fixed elements like system entry points and version numbers. Based on this understanding, this step identifies these stable path level combinations to form the basis for the second grouping, thereby further subdividing the current first URL group into multiple second URL groups. Specifically, for each first URL group, this step intelligently determines the path level combination used for the second grouping by analyzing the distribution characteristics of the deduplication rate of path nodes at different levels within each URL path.

[0078] Step S204: Perform cluster analysis on the path nodes within each second URL group to identify wildcard path nodes.

[0079] In this context, a wildcard path refers to an API path pattern that includes variables represented by placeholders. For example, ` / api / v1 / users / 123` and ` / api / v1 / users / 456` can be abstracted as the wildcard path ` / api / v1 / users / {user_id}`, where `{user_id}` is the placeholder variable. Clustering analysis is an unsupervised machine learning method that aims to automatically group samples with similar characteristics into the same group. In this scheme, by extracting the features of path nodes and analyzing their similarity, path node values ​​representing the same type of wildcard parameter (such as all numeric IDs) can be automatically grouped into one cluster, thereby identifying wildcard path nodes.

[0080] Specifically, for each second URL group, the characteristics of its path nodes are analyzed, and cluster analysis is performed on the path nodes. Based on the results of the cluster analysis, path nodes belonging to the same cluster are identified as wildcard path nodes that follow the same wildcard pattern. For example, if the cluster analysis groups the path node values ​​"123", "456", and "789" into the same cluster, these nodes are identified as wildcard parameters of the same type, and their corresponding path levels are marked as wildcard path nodes.

[0081] Step S205: Generate a list of wildcard path API assets based on wildcard path nodes.

[0082] The API asset list refers to a standardized list that includes all identified wildcard paths and is used to comprehensively reflect the status of the system's API interfaces.

[0083] Specifically, based on the wildcard path nodes identified in the above steps, the corresponding node positions in the original URL paths are replaced with a unified wildcard identifier (such as {param}). All generated wildcard paths are then deduplicated and merged to ultimately generate a complete, standardized, and non-redundant list of API assets using wildcard paths. For example, if the original URL paths are / api / v1 / users / 123, / api / v1 / users / 456, and / api / v1 / products / abc123, after wildcard path identification and replacement, their corresponding API asset list will be [" / api / v1 / users / {param}", " / api / v1 / products / {param}"]. In this way, a large number of specific URL instances are abstracted into a few representative wildcard paths, forming a concise and complete API asset list.

[0084] Through steps S201 to S205, the data environment is refined through two-level grouping. Simultaneously, wildcard path nodes are accurately identified through feature analysis and clustering, enabling fully automated and highly accurate extraction of wildcard path API assets from massive and chaotic network traffic. Compared to URL prefix tree-based analysis methods, which are prone to path combination explosion, implementation complexity, and limited support for complex wildcard patterns, this solution combines grouping and clustering. First, massive URL paths are intelligently divided into multiple structurally similar, small-scale second URL groups through two groupings. Since the URL paths within each group have the same stable prefix structure and depth, the dimension and scale of their path node matrix are significantly reduced. This decomposes the global, high-dimensional complex computation into multiple clustering computation tasks performed on finite, homogeneous subsets of data. This "divide and conquer" strategy fundamentally avoids the exponential increase in computational complexity caused by dynamic expansion and global traversal in traditional tree structures when processing wildcard paths, significantly reducing overall computational complexity. Furthermore, this significant reduction in computational complexity directly leads to a substantial reduction in implementation difficulty. Because the data within each second URL group has a highly consistent structure, the clustering analysis module designed for it can be highly standardized and reusable, eliminating the need for complex dynamic tree structure maintenance and traversal algorithms to handle globally diverse path combinations. This makes the core algorithm's logic clearer and more stable, reducing the burden of code implementation and system maintenance. Simultaneously, the multiple independent computing units formed after grouping are easier to deploy and process in parallel, further simplifying the design and optimization of the system architecture. Furthermore, compared to existing methods relying on manual collection, code instrumentation, or single-rule matching, this solution offers significant advantages: high timeliness, authentic and valid assets, no development intervention required, no dependence on a specific technology stack, and intelligent handling of complex wildcard paths, effectively addressing the core pain points of enterprise API asset management.

[0085] In some embodiments, for each first URL group, the grouping criteria are determined based on the number of duplicates of path nodes in the current first URL group at each path depth level in the corresponding URL path, including:

[0086] Within the current first URL group, each URL path is parsed into a first URL path array consisting of path node values. The first path node matrix is ​​constructed with the first URL path array as the rows and the path depth level of the current first URL group as the columns.

[0087] Calculate the number of duplicates for each column of path nodes in the first path node matrix to obtain the first deduplication count array. Each element in the first deduplication count array corresponds to the number of duplicates for each column of the first path node matrix.

[0088] Calculate the baseline value based on the first deduplication count array;

[0089] By sequentially traversing the first deduplication array, a continuous sequence of elements starting from the first element of the first deduplication array is identified, wherein the value of each element in the continuous sequence is not greater than the pivot value.

[0090] The columns of the first path node matrix corresponding to the continuous element sequence are determined as the grouping key columns;

[0091] The path level corresponding to the grouping key column is determined as the grouping basis.

[0092] In this scheme, a matrix is ​​used to store the structured node information (i.e., path nodes) of all URL paths, and an array is used to record the statistical characteristics (i.e., the number of duplicates) of path nodes at each level. By leveraging the efficiency and standardization of matrix and numerical calculations, a fast and unified processing of massive URL paths is achieved.

[0093] Specifically, within the current first URL group, each URL path is parsed into a first URL path array composed of path node values. A first path node matrix is ​​constructed, with the first URL path array as rows and the path depth level of the current first URL group as columns. The number of rows in this first path node matrix is ​​equal to the number of all URL paths within the current first URL group. The number of unique path nodes in each column of the first path node matrix is ​​calculated, resulting in a first deduplication count array. Each element in the first deduplication count array corresponds to the number of unique nodes in each column of the first path node matrix. Based on the first deduplication count array, a baseline value is calculated. By sequentially traversing the first deduplication count array, a continuous sequence of elements starting from the first element of the first deduplication count array is identified, where the value of each element in the continuous sequence is not greater than the baseline value. The columns of the first path node matrix corresponding to the continuous element sequences are determined as grouping key columns. The path levels corresponding to the grouping key columns are determined as the grouping criteria.

[0094] Further, based on the first deduplication count array, a baseline value is calculated, including:

[0095] Calculate the mean and median of all elements in the first deduplication array to obtain the first mean and the first median;

[0096] The minimum of the first average and the first median is determined as the benchmark value.

[0097] The following example will illustrate the specific details of the above solution.

[0098] Specifically, taking the set of URL paths containing api.example.com / api / v1 / users / 123, api.example.com / api / v1 / users / 456, auth.example.com / oauth / token, and gateway.example.com / v2 / orders / 789 as an example, firstly, through the first grouping (based on domain name and path depth), this set is divided into three first URL groups, namely:

[0099] The group with domain name api.example.com and depth 4 is [api / v1 / users / 123, api / v1 / users / 456];

[0100] The domain name is auth.example.com, and the group at depth 2 is [oauth / token].

[0101] The group with the domain name gateway.example.com and a depth of 3 [v2 / orders / 789];

[0102] For the first URL group at depth 4 of api.example.com, the first path node matrix is ​​constructed as follows:

[0103] .

[0104] Calculate the number of duplicates in each column of the first path node matrix to obtain the first array of duplicate counts [1, 1, 1, 2].

[0105] Calculate the mean and median of all elements in the first deduplication array to obtain the first mean and the first median. That is, the first mean is (1+1+1+2) / 4=1.25 and the first median is (1+1) / 2=1. The baseline value is min(1.25,1)=1.

[0106] Next, iterate through the elements in the first deduplication array sequentially, starting from the first element, and compare the current element value with the pivot value one by one:

[0107] The 0th element has a value of 1 ≤ 1, which satisfies the condition;

[0108] The first element has a value of 1 ≤ 1, which satisfies the condition;

[0109] The second element has a value of 1 ≤ 1, which satisfies the condition;

[0110] The third element has a value of 2 > 1, which does not meet the condition, so the iteration stops.

[0111] Then, the 0th, 1st, and 2nd columns of the first path node matrix corresponding to all the elements that have been compared before the traversal stops (i.e., the 0th, 1st, and 2nd elements) are selected as the grouping key columns. That is, the 0th, 1st, and 2nd levels corresponding to "api", "v1", and "users" are determined as the grouping levels, and the grouping basis is determined. Based on the above grouping basis, the first URL group is maintained as a second URL group: [api / v1 / users / 123, api / v1 / users / 456].

[0112] Taking another URL group containing api / v1 / test / A100001, api / v1 / test / A100005, api / v1 / prd / A100002, api / v2 / prd / A100003, and api / v2 / dev / A100004 as an example, the first path node matrix is ​​constructed as follows:

[0113] .

[0114] Calculate the number of duplicates in each column of the first path node matrix to obtain the first array of duplicate counts [1, 2, 3, 5].

[0115] Calculate the mean and median of all elements in the first deduplication array. The first mean is (1+2+3+5) / 4=2.75, the first median is (2+3) / 2=2.5, and the benchmark value is min(2.75,2.5)=2.5.

[0116] Iterate through the first deduplication array in sequence:

[0117] The 0th element is 1 ≤ 2.5, which satisfies the condition.

[0118] The first element, 2 ≤ 2.5, satisfies this condition;

[0119] The second element 3 > 2.5, which does not satisfy the condition, so the iteration stops.

[0120] Select columns 0 and 1 as grouping keys to determine the 0th and 1st levels corresponding to "api" and version number ("v1" / "v2") as the grouping criteria. Based on this grouping criteria, the first URL group is divided into two second URL groups: group1 [api / v1 / test / A100001, api / v1 / test / A100005, api / v1 / prd / A100002] and group2 [api / v2 / prd / A100003, api / v2 / dev / A100004].

[0121] Through the above process, the grouping criteria are automatically determined based on the statistical characteristics of the number of duplicate path nodes, which enables the intelligent subdivision of a large set of URLs into multiple groups with the same stable prefix. This creates favorable conditions for subsequent accurate clustering analysis and effectively avoids the path combination explosion problem that may occur with traditional tree analysis methods.

[0122] In some embodiments, cluster analysis is performed on path nodes within each second URL group to identify wildcard path nodes, including:

[0123] Within the current second URL group, each URL path is parsed into a second URL path array consisting of path node values. The second URL path array is used as the rows, and the path depth level of the current second URL group is used as the columns to construct a second path node matrix.

[0124] Calculate the number of duplicates for each column of the path nodes in the second path node matrix to obtain the second deduplication count array. Each element in the second deduplication count array corresponds to the number of duplicates for each column of the second path node matrix.

[0125] Based on the preset dimensionality reduction rules, the dimensionality reduction of each path node value in the second path node matrix is ​​performed to obtain the dimensionality reduction array corresponding to each path node value.

[0126] For each column in the second path node matrix, calculate the statistical characteristics of the current column. The statistical characteristics include the second number of duplicates and the second average value of the current column. The second average value is the numerical average value calculated based on the dimensionality reduction array of all path node values ​​in the current column.

[0127] Based on the second deduplication count array and the second average value of each column, the target columns that need to be clustered are marked from all columns of the second path node matrix;

[0128] Clustering algorithms are applied to the target columns to perform cluster analysis, and the cluster analysis results are obtained.

[0129] Based on the clustering analysis results, path nodes within the same cluster are identified as wildcard path nodes under the same wildcard pattern.

[0130] Specifically, in the second URL grouping, the string-form path nodes are converted into numerical features through dimensionality reduction, and then the target columns most likely to contain wildcard parameters are selected through statistical feature analysis. Finally, similar patterns are clustered into one class through a clustering algorithm, and path nodes belonging to the same cluster are identified as wildcard path nodes under the same wildcard pattern.

[0131] Furthermore, based on preset dimensionality reduction rules, dimensionality reduction processing is performed on each path node value in the second path node matrix, including:

[0132] Type identification is performed on the current path node values ​​in the second path node matrix;

[0133] If the path node value is identified as a predefined special type, the current path node value will be mapped to a preset fixed numerical array. The special type includes at least one of the following: pure number, UUID, hexadecimal string, and file resource name.

[0134] If it is not identified as a special type, then according to the predefined character mapping table, each character in the string of the current path node value is converted into the corresponding numerical value, and the current path node value is formed into a reduced-dimensional array in order;

[0135] The character mapping table defines different numerical mapping relationships for different character sets.

[0136] Specifically, pure numbers, UUIDs, hexadecimal strings, and file resource names are reduced to a fixed array to force them into a single category during clustering; other characters are reduced in dimensionality according to a preset mapping table. The purpose of this dimensionality reduction is to convert characters into numerical values, preparing data for subsequent clustering analysis.

[0137] It is understandable that the preset dimensionality reduction rules are designed based on the characteristics of different data paradigms. In a preferred embodiment, the dimensionality reduction rules can be preset as follows:

[0138] Pure numeric types are mapped as [1], UUID types as [2], hexadecimal strings as [3], file resource names as [4], and other types are reduced in dimensionality according to the character mapping table shown in Table 1:

[0139] Table 1

[0140]

[0141] Similarly, it can be understood that the above rules for marking target columns can be implemented in the following preferred embodiment:

[0142] The current column in the second path node matrix is ​​identified as the target column and marked when it meets at least one of the following rules:

[0143] Rule 1: If the path depth corresponding to the current second URL group is 1;

[0144] Rule 2: If the second average value of the current column is less than the preset clustering threshold;

[0145] Rule 3: The second deduplication count in the current column is greater than the median of the second deduplication count array, and is greater than or equal to the sum of the average value and the adjustment value of the second deduplication count array;

[0146] The adjustment value is calculated as follows: the standard deviation of the second deduplication array is multiplied by a preset coefficient, and then multiplied by the serial number of the second deduplication value in the current column in the second deduplication array in ascending order of numerical value.

[0147] In detail, the specific rules for marking the target column are as follows:

[0148] Rule 1: If there is only one URL path in the second URL group, cluster analysis must be performed.

[0149] Rule 2: If the second average value of the current column is less than the preset clustering threshold, it means that the path node values ​​of the column are mainly composed of character types (such as numbers and lowercase letters) with low numerical characteristics after dimensionality reduction, and there is a high probability of wildcard parameters. Therefore, it is marked as a target column that needs to be clustered. The clustering threshold is set according to the numerical characteristics of character mapping in the specific dimensionality reduction rule.

[0150] Rule 3: The second deduplication count in the current column is greater than the median of the second deduplication count array, and is greater than or equal to the sum of the average value and the adjustment value of the second deduplication count array.

[0151] The purpose of this rule is to accurately locate the most likely wildcard parameter column. In RESTful API design, the number of duplicates (i.e., different parameter values) at the path level where wildcard parameters (such as user ID, order number) reside will far exceed the number of duplicates at the level where fixed path terms (such as "api", "v1", "users") reside. Rule 3 automatically extracts this "abnormally high" level through statistical methods.

[0152] The judgment condition is: ( )and( ),

[0153] in, This indicates the second number of duplicates to be removed from the current column. This represents the median of the second deduplication array. This represents the average value of the second deduplication array; ,in, This represents the standard deviation of the second array of duplicate counts. Indicates the preset coefficient. This indicates the index (starting from 0) of the current column's deduplication count value in the second deduplication count array, arranged in ascending order.

[0154] Specifically, the specific process of the above steps will be further explained using the examples of group1 and group2 after the above grouping.

[0155] In group1 [api / v1 / test / A100001, api / v1 / test / A100005, api / v1 / prd / A100002], firstly, the second path node matrix is ​​constructed as follows:

[0156] .

[0157] The second deduplication count array [1, 1, 2, 3] is calculated.

[0158] Based on the dimensionality reduction rules described above, the dimensionality of each path node value in the second path node matrix is ​​reduced, resulting in a set of dimensionality reduction results in the following form:

[0159] The dimension reduction of column 0 ["api","api","api"] is: [[2,2,2], [2,2,2], [2,2,2]];

[0160] The first column ["v1","v1","v1"] is reduced to: [[2,1], [2,1], [2,1]];

[0161] The second column ["test","test","prd"] is reduced to: [[2,2,2,2], [2,2,2,2], [2,2,2]];

[0162] The dimension reduction of column 3 ["A100001","A100005","A100002"] is: [[2.2,1,1,1,1,1,1], [2.2,1,1,1,1,1,1], [2.2,1,1,1,1,1,1]].

[0163] Calculate the second average value of each column in the second path node matrix above:

[0164] Column 0: The average of all reduced-dimensional arrays = (2+2+2+2+2+2+2+2+2) / 9 = 2;

[0165] Column 1: Average of all reduced-dimensional arrays = (2+1+2+1+2+1) / 6 = 1.5;

[0166] Column 2: Average of all reduced-dimensional arrays = (2+2+2+2+2+2+2+2+2+2) / 11 = 2;

[0167] Column 3: Average of all reduced-dimensional arrays ≈ 1.17;

[0168] Based on the rules for marking the target column mentioned above, the filtering process is as follows:

[0169] Rule 1: Path depth 4 > 1, not applicable.

[0170] Rule 2: If the preset clustering threshold is set to 2, then the first column (1.5<2) and the third column (1.2<2) satisfy the condition.

[0171] Rule 3: Calculate the second deduplication array [1, 1, 2, 3], where the mean is 1.75, the median is 1.5, the standard deviation is approximately 0.96, and the preset coefficient is 0.8.

[0172] The following judgments are made in the order of columns: Column 0 (the number of duplicates is 1): Condition 1 is not met (1 > 1.5 is false), so Rule 3 is not met.

[0173] Column 1 (duplicate count is 1): Condition 1 is not met (1 > 1.5 is false), therefore rule 3 is not met.

[0174] Column 2 (duplicate count is 2): condition 1 is satisfied (2 > 1.5 is true), the calculated adjustment value is 0.96 × 0.8 × 2 = 1.54. Condition 2 requires 2 ≥ 1.75 + 1.54 = 3.29, which is not satisfied. Therefore, rule 3 is not satisfied.

[0175] Column 3 (with 3 duplicates): condition 1 is met (3 > 1.5 is true), the calculated adjustment value is 0.96 × 0.8 × 3 = 2.3. Condition 2 requires 3 ≥ 1.75 + 2.3 = 4.05, which is not met. Therefore, rule 3 is not met.

[0176] Because rule two is satisfied, the first column (column number starts from 0, i.e., index 1) and the third column (column number starts from 0, i.e., index 3) in the second path node matrix are finally marked as target columns.

[0177] Similarly, for group2 [api / v2 / prd / A100003, api / v2 / dev / A100004], construct the second path node matrix:

[0178] .

[0179] The second deduplication array [1, 1, 2, 2] is calculated. After dimensionality reduction and statistical feature calculation, the third column of the corresponding second path node matrix is ​​marked according to the marking rules (the column number starts from 0, and the third column here is the column with index 3) as the target column.

[0180] By employing the aforementioned technical solutions, dimensionality reduction processing of path node strings transforms unstructured data, which is difficult to compute directly, into standardized numerical features, effectively overcoming the common computational challenges faced by clustering algorithms when processing string data. Building upon this, by introducing special type recognition and character mapping mechanisms, semantically similar parameter values ​​are mapped to identical or similar numerical vectors. This not only ensures that wildcard parameters of the same type remain highly clustered in the feature space but also provides a stable and reliable metric for subsequent distance-based clustering analysis. Furthermore, based on statistical features, and comprehensively utilizing multi-dimensional statistical information such as the distribution of deduplication counts and dimensionality reduction feature values, the system automatically identifies the high-variability path levels most likely to contain wildcard parameters and intelligently marks target columns. This avoids the computational resource consumption associated with clustering all path columns while ensuring that key wildcard path levels are not overlooked. Through the organic combination and synergistic effect of these two key technologies, clustering analysis can focus on preprocessed and optimized data, significantly improving computational efficiency while ensuring the accuracy of wildcard path identification. Ultimately, this technical solution provides solid and reliable technical support for automatically and accurately extracting wildcard path API assets from massive network traffic.

[0181] In some embodiments, a clustering algorithm is applied to the target column to perform clustering analysis, and the clustering analysis results are obtained, including:

[0182] Get the number of path nodes and average character length of the current target column;

[0183] Based on the number of path nodes and the average character length, the neighborhood radius parameter eps and the minimum number of points parameter min_pts of the DBScan algorithm are determined through predefined calculation rules.

[0184] The DBScan algorithm, configured with the above eps and min_pts parameters, is used to perform cluster analysis on the target column, and the cluster analysis results are obtained.

[0185] Among them, the neighborhood radius refers to the maximum distance threshold in the feature space for determining whether two data points belong to the same neighborhood. It is used to characterize the granularity of clustering. The smaller the value, the finer the clustering; the larger the value, the looser the clustering. The minimum number of points refers to the minimum number of data points required to form an effective cluster. It is used to characterize the density requirement of the cluster. The larger the value, the higher the density requirement of the cluster.

[0186] Specifically, the number of path nodes and the average character length of the current target column are obtained; based on the number of path nodes and the average character length, the neighborhood radius parameter eps and the minimum number of points parameter min_pts of the DBScan algorithm are determined through predefined calculation rules; the DBScan algorithm configured with the above eps and min_pts parameters is applied to perform cluster analysis on the target column to obtain the cluster analysis results.

[0187] In this scheme, the DBScan algorithm is optimized by dynamically determining the values ​​of two parameters based on data characteristics, and then applying these parameter values ​​to perform cluster analysis on the target column. In this scheme, eps is positively correlated with the number of path nodes and the average character length, and does not exceed half of the average character length; min_pts is positively correlated with the number of path nodes.

[0188] In detail, the calculation rules for the two parameters of the DBScan algorithm in this scheme are as follows:

[0189] First, the following configuration parameters are predefined. These parameters need to be adjusted according to different business scenarios and data characteristics, and must be pre-configured before the algorithm runs.

[0190] (1) Five configuration parameters related to the neighborhood radius:

[0191] Default neighborhood radius default_eps, large number of baseline neighborhood radius l2_eps, large number of minimum node count threshold min_l2_eps_count, character length baseline value length_eps_interval, character length coefficient length_eps_step;

[0192] (2) Two configuration parameters related to the minimum number of points: the default value of the minimum number of points min_pts_default, and the calculation coefficient of the minimum number of points min_pts_ratio;

[0193] Then, obtain the number of path nodes y and the average character length len_avg of the current target column, and dynamically determine the neighborhood radius parameter eps and the minimum number of points parameter min_pts of the DBScan algorithm according to the following process:

[0194] The process for determining the neighborhood radius parameter eps is as follows:

[0195] If y > min_l2_eps_count, then eps = l2_eps × (y / min_l2_eps_count);

[0196] Otherwise, if y <= min_l2_eps_count and len_avg > lenth_eps_interval, then eps = (len_avg / lenth_eps_interval) × length_eps_step;

[0197] Otherwise, eps = default_eps;

[0198] Ultimately, the value of eps will not exceed len_avg / 2, that is, eps = min(eps, len_avg / 2);

[0199] The process of determining the minimum number of points parameter min_pts is as follows: min_pts = max( y / min_pts_ratio, min_pts_default );

[0200] Through the above steps, the DBScan algorithm can adapt to URL path grouping of different scales and data characteristics through a dynamic parameter configuration mechanism based on data features. This ensures effective clustering even with large amounts of data, while avoiding excessive noise clusters when the amount of data is small. This significantly improves the stability and accuracy of clustering results and provides a reliable guarantee for the accurate identification of wildcard path nodes.

[0201] In some embodiments, the DBScan algorithm configured with the aforementioned eps and min_pts parameters is applied to perform cluster analysis on the target column, and the method further includes:

[0202] During the DBScan algorithm's cluster expansion process, when the number of points in the neighborhood of the current core point first meets the target condition, a subset of points is extracted from the neighborhood point set of the current core point according to a preset sampling rule to form a sample point set. Subsequent neighborhood queries and cluster expansion operations are only performed on the sample point set. The target condition is that the number of points in the neighborhood of the core point is greater than a preset absolute number threshold, and the proportion of the number of points in the neighborhood to the total number of points in the current target column is greater than a preset dense region proportion threshold.

[0203] Among them, the absolute quantity threshold is used to identify large-scale neighborhoods from an absolute scale, and the dense area proportion threshold is used to identify ultra-dense areas from a relative scale. The dual conditions ensure that optimization is enabled only in areas where the data is truly dense.

[0204] Specifically, when expanding a cluster, the standard DBScan algorithm recursively performs neighborhood queries and expansions on all points within the neighborhood of the core point. When dealing with massive amounts of URL path data, certain wildcard parameters (such as numeric IDs) may form extremely large neighborhoods. In this case, the traditional full expansion method will lead to a sharp increase in computational load, becoming a performance bottleneck.

[0205] To address this, this solution introduces a simplified cluster expansion mechanism, setting two thresholds: an absolute number threshold and a relative dense region proportion threshold. When the algorithm first discovers a core point whose number of neighborhood points simultaneously exceeds both the absolute number threshold (e.g., 1000 points) and the relative dense region proportion threshold (e.g., 20% of the total points), the region is determined to be an "ultra-dense region." In this case, the system no longer recursively expands all points in the neighborhood. Instead, it employs a pre-defined sampling strategy, such as sampling a subset of points from the neighborhood point set at fixed intervals (e.g., 1 point out of every 100 points) to form a sample point set. Subsequent neighborhood queries and cluster expansion operations are only performed on these sample points, thus avoiding the technical bottlenecks of traditional full expansion methods.

[0206] Through the above steps, by introducing a sampling expansion mechanism for ultra-dense regions, the number of recursive calculations for neighborhood queries and expansions is significantly reduced without significantly affecting the accuracy of the final clustering results. This greatly improves the computational efficiency of the algorithm when processing massive amounts of URL path data, and solves the performance bottleneck problem of the traditional DBScan algorithm when facing large-scale data clustering.

[0207] In some embodiments, after obtaining the clustering analysis results, the method further includes:

[0208] Based on the number of members in each cluster in the cluster analysis results, and combined with a dynamic threshold, clusters with a number of members not less than the dynamic threshold are selected to obtain the first set of verification results.

[0209] Based on the first set of verification results, and combined with the preset API keyword library, clusters in which the proportion of keywords in the API keyword library contained in the path node values ​​does not exceed the preset keyword proportion threshold are selected to obtain the second set of verification results.

[0210] Based on the clustering analysis results, path nodes within the same cluster are identified as wildcard path nodes under the same wildcard pattern, including:

[0211] Based on the second set of verification results, all path nodes within the clusters in the second set of verification results are identified as wildcard path nodes under the same wildcard pattern.

[0212] The dynamic threshold is determined as follows: obtain the total number of URLs in the current second URL group and the column number of the current target column in the second path node matrix; calculate the ratio of the total number of URLs to an adjustment denominator as the dynamic threshold; wherein the adjustment denominator is the maximum value of 2 and the square of the column number.

[0213] Specifically, after obtaining the cluster analysis results, the method also includes a cluster analysis result verification step. The purpose of this step is to screen and filter the preliminary results generated by the cluster analysis and eliminate unreasonable clusters.

[0214] This solution employs a dual verification mechanism: a threshold verification based on the number of members and a semantic verification based on API keywords.

[0215] The first step is a threshold check based on the number of members. A true wildcard path pattern (such as a user ID) typically corresponds to a large number of different values ​​(i.e., forming a large cluster). If a cluster has too few members, it is likely just a coincidence, a spelling error, or a rarely used parameter, lacking value as a general API asset. Therefore, in this solution, a dynamic threshold is set to intelligently filter statistically significant clusters. This threshold is not fixed but widens as the data volume (total number of URLs) increases and tightens as the path hierarchy (column number) deepens, thus intelligently adapting to groupings of different sizes and data depths. In a preferred embodiment, the dynamic threshold is defined as: Dynamic threshold = Total number of URLs / max(2, column number²), where the total number of URLs is the total number of URLs in the current second URL group (i.e., the number of rows in the second path node matrix), and the column number is the index of the current target column in the second path node matrix (column numbers in the matrix start from 0).

[0216] In the actual verification process, the number of members in each cluster in the cluster analysis results (i.e. the number of path node values ​​contained in the cluster) is calculated. The calculated number of members is compared with the above dynamic threshold. Clusters with a number of members not less than the dynamic threshold are retained; clusters with a number of members less than the threshold are filtered out, thereby effectively filtering out small clusters or noise points that are generated by chance and lack universality.

[0217] Next, semantic validation is performed based on API keywords. If a cluster contains a large number of values ​​that are fixed keywords such as "get", "list", and "query", then this cluster likely represents not a single parameter (e.g., / api / v1 / users / {id}), but rather a series of different fixed endpoints (e.g., / api / v1 / users / get, / api / v1 / users / list). Merging them into a single wildcard path / api / v1 / users / {param} is clearly incorrect. This validation is precisely to intercept such erroneous merging.

[0218] Specifically, for clusters that pass the first verification (i.e., the "first verification result set"), it is checked whether the path node value (original string) of each member within the cluster appears in a pre-defined API keyword library. This API keyword library is a pre-defined list containing a large number of common API fixed path terms (such as: user, api, get, list, id, query, etc.). The proportion of members matching the keyword library in the entire cluster is calculated relative to the total number of members in the cluster. Clusters with this proportion not exceeding a preset keyword proportion threshold (e.g., 30%) are retained; clusters with a proportion exceeding this threshold are filtered out, thereby effectively preventing semantic terms that should be used as fixed paths from being incorrectly identified as wildcard parameters.

[0219] Through the above steps, by setting up a dual verification mechanism, the clustering results are strictly screened from two dimensions: statistical significance and semantic rationality. This ensures the general representativeness of wildcard path patterns and avoids misidentifying fixed paths as wildcard parameters, significantly improving the accuracy and usability of the final API asset list and providing reliable protection for enterprise API asset management and security auditing.

[0220] In some embodiments, a list of wildcard path API assets is generated based on wildcard path nodes, including: replacing wildcards in the original URL path with the identified wildcard path nodes to generate corresponding wildcard path templates; deduplicating all generated wildcard path templates; and organizing the deduplicated wildcard path templates into a standardized API asset list.

[0221] Specifically, in a preferred embodiment, the API asset list can be obtained as follows:

[0222] First, based on the wildcard path node identification results obtained after cluster analysis and verification, the specific node positions in each URL path that need to be replaced with wildcards are determined. Then, the corresponding path node values ​​in the original URL paths are replaced with uniform wildcard identifiers (such as {param}, {id}, etc.), generating wildcard path templates. Next, all generated wildcard path templates are globally deduplicated to eliminate duplicates. Finally, the deduplicated wildcard path templates are sorted and organized according to specific rules (such as lexicographical order, path depth, etc.) to form the final wildcard path API asset list.

[0223] The implementation process is illustrated below with specific examples:

[0224] First, wildcard replacement is performed. Taking the aforementioned group1 group as an example, the path node in the 3rd column (index 3) is identified as a wildcard path node. For the URL paths api / v1 / test / A100001 and api / v1 / test / A100005, their 3rd path node is replaced with a wildcard, generating a wildcard path template: api / v1 / test / {param}.

[0225] Next, path restoration and template generation are performed. Group1 contains the following URL paths:

[0226] api / v1 / test / A100001;

[0227] api / v1 / test / A100005;

[0228] api / v1 / prd / A100002;

[0229] After wildcard replacement, the following are generated:

[0230] api / v1 / test / {param};

[0231] api / v1 / test / {param};

[0232] api / v1 / prd / {param}.

[0233] Next, deduplication is performed. The generated wildcard path template is deduplicated to obtain:

[0234] api / v1 / test / {param};

[0235] api / v1 / prd / {param}.

[0236] Similarly, for group2, the wildcard path obtained through the above steps is as follows:

[0237] api / v2 / prd / {param};

[0238] api / v2 / dev / {param}.

[0239] Finally, the API asset list is generated. All deduplicated wildcard path templates are organized into the API asset list:

[0240] ["api / v1 / test / {param}","api / v1 / prd / {param}","api / v2 / prd / {param}","api / v2 / dev / {param}"].

[0241] Through the above steps, this solution can abstract massive amounts of specific URL instances into concise and representative wildcard path templates through this automated replacement, deduplication, and organization process. It retains the core structural features of the URL while eliminating the interference of specific parameter values. The resulting API asset list is characterized by high integrity, low redundancy, and standardized structure, which greatly facilitates subsequent API asset management, security auditing, and monitoring analysis.

[0242] In practical applications, the entire API asset identification process involves multiple data processing stages. To more clearly illustrate the data structure and processing logic of each stage, the following section explains the implementation of the technical solution through key data structure definitions and algorithm implementations.

[0243] First, in the path feature analysis phase, the system uses specific data structures to store and analyze the feature information of URL paths. Below is an example of a data structure for URL grouping feature analysis:

[0244] {

[0245] / / Domain name of the current group URL

[0246] "domain": "domain name",

[0247] / / URL path depth of the current group

[0248] "x": 4,

[0249] / / Number of URL paths in the current group

[0250] "y": 5,

[0251] / / A two-dimensional array of strings, where each subset of URLs in a group has the same URL path depth, and a two-dimensional data structure records the value of each path node.

[0252] "nodes": [

[0253] ["api", "v1", "test", "A10001"],

[0254] ["api", "v1", "prd", "A10002"],

[0255] ["api", "v2", "prd", "A10003"],

[0256] ["api", "v2", "dev", "A10004"],

[0257] ["api", "v1", "test", "A10005"]

[0258] ],

[0259] / / Array of strings, the raw URL values ​​of the current group (excluding the domain).

[0260] "paths": [

[0261] "api / v1 / test / A100001",

[0262] "api / v1 / test / A100005"

[0263] ],

[0264] / / A one-dimensional array of type objects is used to perform sibling path feature analysis on path nodes with the same depth.

[0265] "yInfos": [

[0266] {

[0267] / / An array of node objects at the same level after removing duplicates based on the path value

[0268] "yNodes": [

[0269] {

[0270] / / Node path value

[0271] "v": "api",

[0272] / / Number of times the value of the current node at the same level appears

[0273] "c": 2,

[0274] / / An array formed by reducing the dimensionality of the node path value characters. The dimensionality reduction rules are described in detail in the specific implementation of this specification.

[0275] / / Summary: Conversion is based on a predefined character mapping table (e.g., numbers -> 1, lowercase letters -> 2, uppercase letters -> 2.2, symbols -> 5, Chinese characters -> 7).

[0276] "l1": [2, 2, 2]

[0277] }

[0278] ],

[0279] / / Analyzing the characteristics of sibling columns

[0280] "statictics": {

[0281] / / The average of the reduced dimensions of all characters in the yNodes l1 attribute

[0282] "l1_avg": 2.0,

[0283] / / yNodes Standard deviation of all character values ​​in the l1 attribute of each node (dimension reduction)

[0284] "l1_std": 0.0,

[0285] / / Average value of yNodes c

[0286] "avg": 2,

[0287] / / Standard deviation of yNodes c values

[0288] "std": 0,

[0289] / / Average length of yNodes l1 reduced dimension value (l1.length is usually the length of the path in characters)

[0290] "len_avg": 3.0,

[0291] / / Standard deviation of the length of the l1-reduced yNodes (l1.length is usually the length of the path in characters)

[0292] "len_std": 0.0

[0293] },

[0294] / / The current path column is the level of the URL, starting from 0.

[0295] "x": 0,

[0296] / / Total length of the current column (sibling paths), equal to the array length of the current URL group.

[0297] "y": 2,

[0298] / / The deduplication length of the current column nodes, which is equivalent to nodes.length

[0299] "distinct_count": 1,

[0300] / / Sort the current column distinct_count value among all columns in the url group

[0301] "sort_index": 0,

[0302] / / Whether the current column needs cluster analysis is automatically assigned based on the following rules:

[0303] / / Rule 1: True if the URL group path depth is 1;

[0304] / / Rule 2: If the average value of the current column after dimensionality reduction is less than the preset threshold, then set to true;

[0305] / / Rule 3: If the number of duplicates in the current column is significantly greater than that in other columns (based on the median, average, and dynamic adjustment value), then the result is true;

[0306] / / If any rule is met, it is marked as true.

[0307] "need_merge": true

[0308] }

[0309] ],

[0310] / / The length of the array consisting of the number of duplicates in all columns should be x.

[0311] "distinct_counts": [1, 2, 3, 5]

[0312] }

[0313] Secondly, in the clustering analysis phase, the system uses an optimized DBScan algorithm for wildcard path identification. The core implementation of the DBScan clustering algorithm is as follows:

[0314] public static <E extends Point <e>> List<List <e>> dbscan(List <e>points, ClusterParams params) {

[0315] int clusterId = 0;

[0316] List<List <e>> clusters = new ArrayList<>();

[0317] for (E point : points) {

[0318] if (!point.visited) {

[0319] point.visited = true;

[0320] / / Get all points in the neighborhood

[0321] List <e>neighbors = getNeighbors(points, point,params.eps);

[0322] if (neighbors.size() < params.min_pts) {

[0323] / / Mark as noise point

[0324] point.cluster = -1;

[0325] } else if (neighbors.size() > params.simple_expand_neighbors_size &&

[0326] neighbors.size() / (double) points.size() >params.simple_expand_neighbors_percent) {

[0327] / / When the number of neighboring nodes is large and accounts for a high percentage of the total number of nodes, create a new cluster to simplify the process.

[0328] List <e>cluster = new ArrayList<>();

[0329] simpleExpandCluster(points, point, neighbors,clusterId, params.eps, params.min_pts, cluster);

[0330] clusters.add(cluster);

[0331] clusterId++;

[0332] } else {

[0333] / / Create a new cluster

[0334] List <e>cluster = new ArrayList<>();

[0335] expandCluster(points, point, neighbors, clusterId,params.eps, params.min_pts, cluster);

[0336] clusters.add(cluster);

[0337] clusterId++;

[0338] }

[0339] }

[0340] }

[0341] return clusters;

[0342] }

[0343] Finally, in the simplified extended cluster processing stage, a sampling optimization strategy is adopted for ultra-dense regions:

[0344] /

[0345] When the number of neighboring nodes is large and accounts for a high percentage of the total number of nodes, creating a new cluster simplifies the sampling process.

[0346] /

[0347] private static <E extends Point <e>> void simpleExpandCluster(List <e>points, E point,

[0348] List <e>neighbors, int clusterId, double eps, int minPts,List <e>cluster) {

[0349] point.cluster = clusterId;

[0350] cluster.add(point);

[0351] Queue <e>queue = new LinkedList<>(neighbors);

[0352] Set <object>added = neighbors.stream().map(Point::id).collect(Collectors.toSet());

[0353] / / 100 is the sampling coefficient, which can be defined by yourself.

[0354] int ratio = neighbors.size() / 100;

[0355] int count = 0;

[0356] while (!queue.isEmpty()) {

[0357] E current = queue.poll();

[0358] if (!current.visited) {

[0359] current.visited = true;

[0360] count++;

[0361] / / Sampling

[0362] if (count % ratio == 0) {

[0363] List <e>currentNeighbors = getNeighbors(points,current, eps);

[0364] if (currentNeighbors.size() >= minPts) {

[0365] / / Only add points that have not yet been added to the queue to avoid multiple loops.

[0366] for (E neighbor : currentNeighbors) {

[0367] if (added.add(neighbor.id())) {

[0368] queue.add(neighbor);

[0369] }

[0370] }

[0371] }

[0372] }

[0373] }

[0374] / / If it was previously a noise point or unclassified

[0375] if (current.cluster == -1) {

[0376] current.cluster = clusterId;

[0377] cluster.add(current);

[0378] }

[0379] }

[0380] }

[0381] It should be noted that the code examples above demonstrate the key implementation details of the technical solution in this application, including:

[0382] (1) Data structure design for path feature analysis;

[0383] (2) The core logic of the DBScan clustering algorithm;

[0384] (3) Simplify the optimization processing mechanism for extended clusters;

[0385] These code examples are intended to help those skilled in the art better understand the specific implementations of the present invention, but should not be construed as limiting the scope of protection of this application. In practical applications, different programming languages, algorithms, or data structures can be used to achieve the same technical effect, and these variations should all fall within the scope of protection of this application.

[0386] Through the complete implementation of the above technical solutions, the system can efficiently and automatically identify wildcard path API assets from massive network traffic, providing reliable technical support for enterprise API security management.

[0387] It should be further noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0388] This embodiment also provides an API asset identification device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0389] Figure 3 This is a structural block diagram of an API asset identification device according to an embodiment of this application, such as... Figure 3 As shown, the device includes: an acquisition module 10, a grouping module 20, and an identification module 30;

[0390] Among them, the acquisition module 10 is used to acquire the set of URL paths in network traffic;

[0391] Grouping module 20 is used to perform the first grouping of the URL path set based on the domain name and path depth of the URL path, resulting in multiple first URL groups;

[0392] Grouping module 20 is also used to determine the grouping criteria for each first URL group based on the number of deduplications of the path nodes in the current first URL group at each path depth level in the corresponding URL path, and to perform a second grouping based on the grouping criteria to obtain multiple second URL groups;

[0393] The identification module 30 is used to perform cluster analysis on the path nodes in each second URL group, identify wildcard path nodes, and generate a wildcard path API asset list based on the wildcard path nodes.

[0394] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0395] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0396] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0397] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0398] S1, retrieves the set of URL paths in network traffic.

[0399] S2, based on the domain name and path depth of the URL path, performs the first grouping of the URL path set, resulting in multiple first URL groups.

[0400] S3. For each first URL group, determine the grouping criteria based on the number of deduplications of the path nodes in the current first URL group at each path depth level in the corresponding URL path, and perform a second grouping based on the grouping criteria to obtain multiple second URL groups.

[0401] S4 performs cluster analysis on the path nodes within each second URL group to identify wildcard path nodes.

[0402] S5 generates a list of wildcard path API assets based on wildcard path nodes.

[0403] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0404] Furthermore, in conjunction with the API asset identification methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the API asset identification methods in the above embodiments.

[0405] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0406] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0407] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0408] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.< / e> < / object> < / e> < / e> < / e> < / e> < / e> < / e> < / e> < / e> < / e> < / e> < / e> < / e>

Claims

1. An API asset identification method, characterized in that, include: Retrieve the set of URL paths from network traffic; Based on the domain name and path depth of the URL path, the set of URL paths is first grouped to obtain multiple first URL groups; For each first URL group, the grouping criteria are determined based on the number of deduplications of the path nodes in the current first URL group at each path depth level in the corresponding URL path, and a second grouping is performed based on the grouping criteria to obtain multiple second URL groups. Cluster analysis is performed on the path nodes within each second URL group to identify wildcard path nodes; Based on the wildcard path nodes, a wildcard path API asset list is generated.

2. The API asset identification method according to claim 1, characterized in that, The method for determining the grouping criteria for each first URL group, based on the number of duplicates of path nodes in the current first URL group at each path depth level in the corresponding URL path, includes: Within the current first URL group, each URL path is parsed into a first URL path array consisting of path node values. A first path node matrix is ​​constructed with the first URL path array as the row and the path depth level of the current first URL group as the column. Calculate the number of duplicates for each column of path nodes in the first path node matrix to obtain a first deduplication count array. Each element in the first deduplication count array corresponds to the number of duplicates for each column of the first path node matrix. Calculate the baseline value based on the first deduplication count array; By sequentially traversing the first deduplication count array, a continuous sequence of elements starting from the first element of the first deduplication count array is identified, wherein the value of each element in the continuous sequence of elements is not greater than the benchmark value; The columns of the first path node matrix corresponding to the continuous element sequence are determined as grouping key columns; The path level corresponding to the grouping key column is determined as the grouping basis.

3. The API asset identification method according to claim 2, characterized in that, The step of calculating the baseline value based on the first deduplication count array includes: Calculate the mean and median of all elements in the first deduplication array to obtain the first mean and the first median; The minimum value between the first average and the first median is determined as the benchmark value.

4. The API asset identification method according to claim 1, characterized in that, The clustering analysis performed on each second URL group to identify wildcard path nodes includes: Within the current second URL group, each URL path is parsed into a second URL path array consisting of path node values. A second path node matrix is ​​constructed with the second URL path array as the rows and the path depth level of the current second URL group as the columns. Calculate the number of duplicates for each column of path nodes in the second path node matrix to obtain a second deduplication count array. Each element in the second deduplication count array corresponds to the number of duplicates for each column of the second path node matrix. Based on the preset dimensionality reduction rules, the dimensionality reduction process is performed on each path node value in the second path node matrix to obtain the dimensionality reduction array corresponding to each path node value; For each column in the second path node matrix, calculate the statistical characteristics of the current column. The statistical characteristics include the second number of duplicates and the second average value of the current column. The second average value is the numerical average value calculated based on the dimensionality reduction array of all path node values ​​of the current column. Based on the second deduplication count array and the second average value of each column, the target columns that need to be clustered are marked from all columns of the second path node matrix; Clustering analysis is performed on the target column using a clustering algorithm to obtain the clustering analysis results; Based on the clustering analysis results, path nodes within the same cluster are identified as wildcard path nodes under the same wildcard pattern.

5. The API asset identification method according to claim 4, characterized in that, The dimensionality reduction process, based on a preset dimensionality reduction rule, performs dimensionality reduction on each path node value in the second path node matrix, including: Type identification is performed on the current path node values ​​in the second path node matrix; If the current path node value is identified as a predefined special type, then the current path node value is mapped to a preset fixed numerical array. The special type includes at least one of pure numbers, UUIDs, hexadecimal strings, and file resource names. If it is not identified as the special type, then according to the predefined character mapping table, each character in the string of the current path node value is converted into the corresponding numerical value, and the current path node value is formed into a reduced-dimensional array in order; The character mapping table defines different numerical mapping relationships for different character sets.

6. The API asset identification method according to claim 4, characterized in that, Based on the second deduplication count array and the second average value of each column, the target columns for clustering analysis are marked from all columns of the second path node matrix, including: The current column in the second path node matrix is ​​identified as the target column and marked when it meets at least one of the following rules: Rule 1: If the path depth corresponding to the current second URL group is 1; Rule 2: If the second average value of the current column is less than the preset clustering threshold; Rule 3: The second deduplication count in the current column is greater than the median of the second deduplication count array, and is greater than or equal to the sum of the average value and the adjustment value of the second deduplication count array; The adjustment value is calculated as follows: the standard deviation of the second deduplication array is multiplied by a preset coefficient, and then multiplied by the serial number of the second deduplication value in the current column in the second deduplication array in ascending order of numerical value.

7. The API asset identification method according to claim 4, characterized in that, The process of applying a clustering algorithm to the target column to perform clustering analysis and obtain the clustering analysis results includes: Get the number of path nodes and average character length of the current target column; Based on the number of path nodes and the average character length, the neighborhood radius parameter eps and the minimum number of points parameter min_pts of the DBScan algorithm are determined through predefined calculation rules. The DBScan algorithm, configured with the parameters eps and min_pts, is used to perform cluster analysis on the target column to obtain the cluster analysis results.

8. The API asset identification method according to claim 7, characterized in that, The application is configured with the DBScan algorithm, using the parameters eps and min_pts, to perform cluster analysis on the target column, and also includes: During the DBScan algorithm's cluster expansion process, when the number of points in the neighborhood of the current core point meets the target condition for the first time, a portion of points are extracted from the neighborhood point set of the current core point according to a preset sampling rule to form a sampling point set, and subsequent neighborhood queries and cluster expansion operations are only performed on the sampling point set. The target conditions are as follows: the number of points in the neighborhood of the core point is greater than a preset absolute number threshold, and the proportion of the number of points in the neighborhood to the total number of points in the current target column is greater than a preset dense area proportion threshold.

9. The API asset identification method according to any one of claims 4 to 8, characterized in that, After obtaining the cluster analysis results, the method further includes: Based on the number of members in each cluster in the clustering analysis results, and combined with a dynamic threshold, clusters with a number of members not less than the dynamic threshold are selected to obtain the first set of verification results. Based on the first set of verification results, and combined with the preset API keyword library, clusters in which the proportion of keywords in the API keyword library contained in the path node values ​​does not exceed a preset keyword proportion threshold are selected to obtain the second set of verification results. The step of identifying path nodes within the same cluster as wildcard path nodes under the same wildcard pattern based on the clustering analysis results includes: Based on the second set of verification results, all path nodes within the clusters in the second set of verification results are identified as wildcard path nodes under the same wildcard pattern. The dynamic threshold is determined as follows: the total number of URLs in the current second URL group and the column number of the current target column in the second path node matrix are obtained; the ratio of the total number of URLs to an adjustment denominator is calculated as the dynamic threshold; wherein the adjustment denominator is the maximum value of 2 and the square of the column number.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the API asset identification method according to any one of claims 1 to 9 at runtime.

Images