URL-based API asset merging method and system

Abstract

The application provides a URL-based API asset merging method and system, which comprises the following steps: obtaining host access record log data; based on the access record log data, extracting a URL, using a specific separator to split the URI, and forming a word set array; according to the word set data, sequentially taking two words as a relationship, taking the words as nodes, constructing a word relationship graph, and calculating the out-degree and in-degree of the words; according to the out-degree and in-degree obtained in the foregoing step, selecting words with out-degree or in-degree less than a specified threshold value by specifying the threshold value; based on the words obtained in the foregoing step, screening out URIs containing the corresponding words, and calculating the URIs with high similarity in the full-amount URIs by using minhash; based on the URIs selected in the foregoing step, calculating the support degree of the selected corresponding words, and replacing the words with a wildcard string if the support degree is less than a specified threshold value, so as to realize API normalization. The application solves the technical problems of a large API asset list and high dependence on manual operation experience.

Description

Technical Field

[0001] This invention relates to the field of data processing and algorithm application technology, specifically to a URL-based API asset merging method and system. Background Technology

[0002] With the development of the internet and mobile internet, API interfaces are widely used in internet business systems. Enterprises provide API interfaces to external users, who then use these interfaces to complete business transactions. As technology and business evolve, enterprises are expanding their openness, increasing the number of API interfaces and API assets. Faced with massive amounts of URL log records, effectively managing and organizing these API assets has become a pressing issue.

[0003] API assets contain numerous API endpoints with highly overlapping paths. These similar API endpoints often share the same purpose. These API endpoints typically only have a fixed path location, differing only in parameters, with the number of path parameter variables reaching hundreds or even thousands. The existing invention patent application document CN115687472A, entitled "A Data Storage Management System Based on a Cloud Platform," includes the following four modules: a hardware resource integration module, which integrates and uniformly manages and evaluates the hardware resources required by various systems; a data transmission and storage module, used for securely transmitting and storing different business data; a data governance module, which cleans and processes the data transmitted to the data transmission and storage module; and a data analysis module, used for analyzing the governed data and outputting the analysis results. From the specific implementation of this prior art, it can be seen that this existing solution classifies different business data according to different labels. This process requires personnel familiar with the business and knowledgeable in data classification and management to sort the data. After data sorting, it is also necessary to supplement data not covered by the business system and correlate it with existing data. The existing literature discloses a solution that categorizes different business data using different labels. This process requires personnel familiar with the business and knowledgeable in data classification and management to organize the data. After the data is organized, it is also necessary to supplement the data not covered by the business system and correlate it with the existing data. Currently, common API asset standardization relies on manual experience and uses methods such as keyword or parameter pattern matching. However, as more and more interfaces are developed, rule matching patterns require a significant investment of manpower to supplement the rules.

[0004] The existing invention patent application document CN112311869A, entitled "Management System and Method for Multi-Type Microservice Registry Centers," includes a registry center, various microservice registry centers, and various application services. Each application service communicates with the microservice registry center according to the original protocol. The registry center includes a server and a client. The server provides service information and routes the registration interface, caches and distributes service and interface information, and supports communication with the client. The client provides an SDK, and each microservice registry center and application service integrates the SDK to communicate with the registry center, report, or subscribe to information. In this prior art, the backend provides the client's registration interface, page management registration service, service list and load balancing configuration, API list, and API interfaces; the frontend displays the page by calling the API interfaces. However, listing all API endpoints with highly overlapping paths and uses could result in an excessively large API asset list.

[0005] In summary, existing technologies suffer from the technical problems of having a large API asset list and being highly dependent on human operational experience. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to address the issue of large API asset lists and high dependence on manual operation experience in existing technologies.

[0007] This invention solves the above-mentioned technical problems by employing the following technical solution: A URL-based API asset merging method includes:

[0008] S1. Obtain host access log data;

[0009] S2. Based on the host access log data, extract the URL, and use a preset delimiter to split the URL into Uniform Resource Identifier (URI) strings to obtain a word set array;

[0010] S3. Based on the word set array, construct a word relationship graph in a preset order according to the word relationships and word nodes in the word set array, and calculate the word out-degree and word in-degree accordingly;

[0011] S4. Based on the word out-degree and word in-degree, and based on the specified out-degree threshold and in-degree threshold, select applicable words whose out-degree is less than the out-degree threshold or whose in-degree is less than the in-degree threshold.

[0012] S5. Using applicable words, filter out applicable Uniform Resource Identifier strings containing corresponding words from the Uniform Resource Identifier strings, and then use the minhash algorithm to calculate similar resource identifier strings in the full set of resource identifier strings.

[0013] S6. Calculate the support of the selected corresponding word based on the similar resource identifier string. When the support is less than the specified support threshold, replace the current word with a wildcard string to merge API assets.

[0014] This invention employs graph relation features and minhash to determine fixed terms in URIs. The calculation process uses the minhash algorithm to filter highly similar interfaces. Based on graph relations, it also analyzes the dynamic nature of terms to ensure the accuracy of dynamic judgment. This invention achieves automatic dynamic term recognition, eliminating the need for human intervention in the API asset management process. It requires neither excessive manual intervention nor the maintenance of a keyword database based on expert experience, thus exhibiting high adaptability to API assets across different industries.

[0015] In a more specific technical solution, step S3 includes a directed relation graph.

[0016] This invention uses graph relationships to measure the importance of terms in industry URIs. As business interfaces continue to increase, there is no need to continuously supplement the rule base, thus improving the efficiency of API interface asset management.

[0017] In a more specific technical solution, step S5 includes:

[0018] S51. Select a specific word V based on the applicable words;

[0019] S52. From the full set of Uniform Resource Identifier (URI) strings, match the applicable Uniform Resource Identifier (URI) strings containing the specific word V.

[0020] S53. Based on the applicable Uniform Resource Identifier (URI) string, in the preset dataset U1, use the minhash algorithm to filter out similar resource identifier strings U2 that have a similarity greater than a preset similarity threshold to the preset dataset U1.

[0021] In a more specific technical solution, step S53 includes:

[0022] S531. Using the minimum hash algorithm minhash, the similarity of the pre-set dataset U1 is obtained based on the Jaccard similarity coefficient.

[0023] S532. Obtain the minimum hash value relationship data of the elements based on the Jaccard similarity coefficient, and then filter out the similar resource identifier string U2.

[0024] This invention employs the minhash algorithm to obtain the similarity of dataset U1 based on the Jaccard similarity coefficient. It uses the similarity to merge API endpoints with overlapping paths and uses, eliminating the need to list all API endpoints and simplifying the size of the API asset list.

[0025] In a more specific technical solution, in step S531, the Jaccard similarity coefficient is determined using the following logic:

[0026]

[0027] In more specific technical solutions, the Jaccard similarity coefficient can take values ​​ranging from (0,1).

[0028] In a more specific technical solution, step S532 uses the following logic to represent the minimum hash value relationship data:

[0029] P[hmni(X)=hmin(y)]=Jaccard(X,y)

[0030] Here, hmni(X) is the element with the smallest hash value in set X, and hmin(Y) is the element with the smallest hash value in set Y.

[0031] In a more specific technical solution, step S5 includes:

[0032] S51. Based on the URI set in the preset dataset U1 and the similar resource identifier string U2, calculate the number of datasets n1 in the preset dataset U1 and the number of identifier strings n2 in the similar resource identifier string U2.

[0033] S52. Calculate the support for a specific word V based on the number of datasets n1 and the number of identifier strings n2.

[0034] S53. When the support is less than the preset support threshold, determine that the specific word V is a variable word and replace the specific word V with a wildcard string.

[0035] This invention uses a minimum hash algorithm to calculate the minimum hash relationship data, and then obtains the word support for judgment. The API asset standardization operation avoids the reliance on human experience judgment, and replaces the keyword or parameter pattern matching method in traditional technology. Even if the scale of interface development increases, no additional manpower is required, which improves the automation and efficiency of API asset standardization.

[0036] In a more specific technical solution, step S52 involves calculating the support level for a specific word V using the following logic:

[0037] n1 / (n1+n2)

[0038] In a more specific technical solution, a URL-based API asset merging system includes:

[0039] The log acquisition module is used to acquire host access record log data;

[0040] The term set acquisition module is used to extract URLs based on host access log data, and split the URLs into Uniform Resource Identifier (URI) strings using preset delimiters to obtain a term set array. The term set acquisition module is connected to the log acquisition module.

[0041] The word in-degree and out-degree calculation module is used to construct a word relationship graph based on the word set array, in a preset order, according to the word relationships and word nodes in the word set array, and to calculate the word out-degree and word in-degree. The word in-degree and out-degree calculation module is connected to the word set acquisition module.

[0042] The applicable word selection module is used to select applicable words whose out-degree is less than the out-degree threshold or whose in-degree is less than the in-degree threshold, based on the word out-degree and word in-degree, and according to the specified out-degree threshold and in-degree threshold. The applicable word selection module is connected to the word out-degree and in-degree calculation module.

[0043] The minimum hash processing module is used to filter out applicable URI strings containing corresponding words from the URI strings using applicable words, and then use the minimum hash algorithm minhash to calculate similar URI strings in the full URI strings. The minimum hash processing module is connected to the applicable word selection module.

[0044] The merge processing module is used to calculate the support of the corresponding words based on similar resource identifier strings. When the support is less than the specified support threshold, the current word is replaced with a wildcard string to merge API assets. The merge processing module is connected to the minimum hash processing module.

[0045] Compared with existing technologies, this invention has the following advantages: This invention uses graph relationship features and minhash to determine fixed words in URIs. The calculation process applies the minhash algorithm to filter highly similar interfaces. Based on graph relationships, it performs dynamic word analysis to ensure the accuracy of dynamic judgment. This invention achieves automatic dynamic word recognition. The API asset sorting process requires no human intervention, eliminating the need for excessive manual intervention or maintenance of keyword databases based on expert experience. Therefore, it has high adaptability to API assets across different industries.

[0046] This invention uses graph relationships to measure the importance of terms in industry URIs. As business interfaces continue to increase, there is no need to continuously supplement the rule base, thus improving the efficiency of API interface asset management.

[0047] This invention employs the minhash algorithm to obtain the similarity of dataset U1 based on the Jaccard similarity coefficient. It uses the similarity to merge API endpoints with overlapping paths and uses, eliminating the need to list all API endpoints and simplifying the size of the API asset list.

[0048] This invention uses a minimum hash algorithm to calculate the minimum hash relationship data, and then obtains the word support for judgment. The API asset standardization operation avoids the reliance on human experience judgment, and replaces the keyword or parameter pattern matching method in traditional technology. Even if the scale of interface development increases, no additional manpower is required, which improves the automation and efficiency of API asset standardization.

[0049] This invention solves the technical problems of existing technologies, such as the large size of API asset lists and the high dependence on manual operation experience. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the basic steps of a URL-based API asset merging method according to Embodiment 1 of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the principle of URI segmentation to form a word set array in Embodiment 1 of the present invention;

[0052] Figure 3 This is a word relationship diagram of Embodiment 1 of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] like Figure 1 As shown, the present invention provides a URL-based API asset merging method, which includes the following basic steps:

[0056] S1. Obtain host access log data;

[0057] In this embodiment, host access log data is extracted within a specified period. In this embodiment, the log content includes, but is not limited to, URLs, request bodies, and response bodies.

[0058] S2. Based on the access log data, extract the URLs, use preset delimiters as delimiters, and segment the URLs to form a word set array; in this embodiment, the preset delimiters include, but are not limited to, " / ";

[0059] like Figure 2As shown, in this embodiment, based on the data obtained in step S1, the URI is segmented using " / " as the delimiter to form a word set array. In this embodiment, for example, if the URI is / news / 1d4d4d5s4ds47we4 / log / , then after segmentation by " / ", the resulting relational combinations are as follows:

[0060] ((news,1d4d4d5s4ds47we4),(1d4d4d5s4ds47we4,log)).

[0061] S3. Based on the word set data, construct a word relationship graph by treating each pair of words as a relationship and each word as a node, and calculate the out-degree and in-degree of each word.

[0062] like Figure 3 As shown, in this embodiment, a directed relation graph is established based on the word combination in step S2, and the out-degree and in-degree of each word are calculated based on the directed relation.

[0063] In this embodiment, let D =<V,E> For a directed graph, v∈V, the out-degree of v is the sum of the number of times v is the starting point of an edge, and the in-degree of v is the sum of the number of times v is the ending point of an edge.

[0064] S4. Based on the out-degree and in-degree obtained in step S3, select words whose out-degree or in-degree is less than the specified threshold by using a specified threshold.

[0065] S5. Based on the words whose out-degree or in-degree is less than the specified threshold selected in step S4, filter out the URIs containing the corresponding words, and use minhash to calculate the URIs with high similarity among all URIs.

[0066] In this embodiment, based on the words filtered in step S4, a word V is selected, and URIs containing word V are matched from the full set of URIs. According to the selected URIs, the dataset U1 is used, and the minhash algorithm is used to filter out U2, which has a similarity greater than 0.8 with U1.

[0067] Minhash is based on the following principles:

[0068] MinHash is a method for quickly detecting the similarity between two sets. MinHash is based on the Jaccard similarity metric. For two sets X and Y, the Jaccard similarity coefficient can be defined as:

[0069]

[0070] In this embodiment, the Jaccard similarity coefficient ranges from, but is not limited to, [0, 1]. The closer two sets are, the closer the value is to 1; conversely, the closer they are to 0. Assume h is a hash function that maps the number of elements in A and B to an integer. Define hmni(S) as the element in set S with the minimum hash value. It can be deduced that hmni(X) = hmin(Y) only if the element with the minimum hash value in X∪Y is located in A∩B. Since the hash function is considered a random variable, any element in X∪Y could potentially have the minimum hash value. Therefore, we have:

[0071] P[hmni(X)=hmin(Y)]=Jaccard(X,Y)

[0072] In this embodiment, if we let r be a random variable, or a function of random variable h, taking the value 1 when hmni(X) = hmin(Y), and 0 otherwise, then r is an unbiased estimate of Jaccard(X,Y).

[0073] S6. Based on the URI selected in step S5, calculate the support of the corresponding word. If the support is less than the specified threshold, replace the word with a wildcard string to achieve API standardization.

[0074] In summary, this invention uses graph relation features and minhash to determine fixed words in URIs. The calculation process uses the minhash algorithm to filter highly similar interfaces. Based on graph relations, it performs dynamic word analysis to ensure the accuracy of dynamic judgment.

[0075] This invention achieves automatic dynamic word recognition, and the API asset sorting process requires no human intervention. It does not require excessive manual intervention or maintenance of a keyword database based on expert experience, thus it has high adaptability to API assets in different industries.

[0076] This invention uses graph relationships to measure the importance of terms in industry URIs. As business interfaces continue to increase, there is no need to continuously supplement the rule base, thus improving the efficiency of API interface asset management.

[0077] This invention employs the minhash algorithm to obtain the similarity of dataset U1 based on the Jaccard similarity coefficient. It uses the similarity to merge API endpoints with overlapping paths and uses, eliminating the need to list all API endpoints and simplifying the size of the API asset list.

[0078] This invention uses a minimum hash algorithm to calculate the minimum hash relationship data, and then obtains the word support for judgment. The API asset standardization operation avoids the reliance on human experience judgment, and replaces the keyword or parameter pattern matching method in traditional technology. Even if the scale of interface development increases, no additional manpower is required, which improves the automation and efficiency of API asset standardization.

[0079] This invention solves the technical problems of existing technologies, such as the large size of API asset lists and the high dependence on manual operation experience.

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A URL-based API asset merging method, characterized in that, The method includes: S1. Obtain host access log data; S2. Based on the host access log data, extract the URL, and use a preset delimiter to segment the URL into a Uniform Resource Identifier (URI) string to obtain a word set array; S3. Based on the word set array, construct a word relationship graph in a preset order according to the word relationships and word nodes in the word set array, and calculate the word out-degree and word in-degree accordingly; S4. Based on the word out-degree and the word in-degree, and based on the specified out-degree threshold and in-degree threshold, select applicable words whose word out-degree is less than the out-degree threshold or whose word in-degree is less than the in-degree threshold. S5. Using the applicable words, filter out applicable Uniform Resource Identifier strings containing the corresponding words from the Uniform Resource Identifier strings, and use the minhash algorithm to calculate similar resource identifier strings in the full set of resource identifier strings. S5 includes: S51. Select a specific word V based on the applicable words; S52. From the URIs of the full set of Uniform Resource Identifiers (URIs), match the URIs of the applicable Uniform Resource Identifier (URI) containing the specific word V. S53. Based on the URI of the applicable Uniform Resource Identifier string, in the preset dataset U1, use the minhash algorithm to filter out similar resource identifier strings U2 that have a similarity greater than the preset similarity threshold with the preset dataset U1. The similarity of the preset dataset U1 is obtained by using the minhash algorithm and processing based on the Jaccard similarity coefficient. The Jaccard similarity coefficient is determined using the following logic: The minimum hash value relationship data of the elements is obtained based on the Jaccard similarity coefficient, and the similar resource identifier string U2 is selected accordingly. The minimum hash value relational data can be represented using the following logic: wherein, is the element in the set with the smallest hash value, is the element in the set with the smallest hash value; S6. Calculate the support of the selected corresponding word based on the similar resource identifier string. When the support is less than the specified support threshold, replace the current word with a wildcard string to merge API assets.

2. The URL-based API asset merging method according to claim 1, characterized in that, In step S3, the word relationship graph includes a directed relationship graph.

3. The URL-based API asset merging method of claim 1, wherein, The Jaccard similarity coefficient ranges from (0, 1).

4. The URL-based API asset merging method of claim 1, wherein, Step S5 includes: S51. Based on the URI set in the preset dataset U1 and the similar resource identifier string U2, calculate the number of datasets n1 in the preset dataset U1 and the number of identifier strings n2 in the similar resource identifier string U2. S52. Calculate the support level of a specific word V based on the number of datasets n1 and the number of identifier strings n2. S53. When the support is less than the preset support threshold, the specific word V is determined to be a variable word, and the specific word V is replaced with a wildcard string.

5. The URL-based API asset merging method of claim 4, wherein, In step S52, the support level containing the specific word V is calculated using the following logic: n1 / (n1+n2) .

6. A URL-based API asset merging system for performing a URL-based API asset merging method according to any one of the preceding claims 1 to 5, characterized in that, The system includes: The log acquisition module is used to acquire host access record log data; The word set acquisition module is used to extract URLs based on the host access log data, and split the URLs into Uniform Resource Identifier strings (URIs) using a preset delimiter to obtain a word set array. The word set acquisition module is connected to the log acquisition module. The word in-degree and out-degree calculation module is used to construct a word relationship graph based on the word set array, in a preset order, according to the word relationships and word nodes in the word set array, and to calculate the word out-degree and word in-degree. The word in-degree and out-degree calculation module is connected to the word set acquisition module; wherein, a specific word V is selected according to the applicable words. From the URIs of the full set of Uniform Resource Identifiers (URIs), match the URIs of the applicable Uniform Resource Identifier (URI) that contain the specific word V; Based on the URIs that use the Uniform Resource Identifier (URI) string, in the preset dataset U1, the minhash algorithm is used to filter out similar resource identifier strings U2 that have a similarity greater than a preset similarity threshold to the preset dataset U1. The similarity of the preset dataset U1 is obtained by using the minhash algorithm and processing based on the Jaccard similarity coefficient. The Jaccard similarity coefficient is determined using the following logic: The minimum hash value relationship data of the elements is obtained based on the Jaccard similarity coefficient, and the similar resource identifier string U2 is selected accordingly. The minimum hash value relational data can be represented using the following logic: wherein, is the element in the set with the smallest hash value, is the element in the set with the smallest hash value; The applicable word selection module is used to select applicable words whose out-degree is less than the out-degree threshold or whose in-degree is less than the in-degree threshold based on the word out-degree, the word in-degree, and a specified out-degree threshold and in-degree threshold. The applicable word selection module is connected to the word out-degree and in-degree calculation module. The minimum hash processing module is used to use the applicable words to filter out applicable Uniform Resource Identifier strings containing the corresponding words from the Uniform Resource Identifier strings, and to use the minimum hash algorithm minhash to calculate similar resource identifier strings in the full set of resource identifier strings. The minimum hash processing module is connected to the applicable word selection module. The merging processing module is used to calculate the support of the corresponding word based on the similar resource identifier string. When the support is less than a specified support threshold, the current word is replaced with a wildcard string to merge API assets. The merging processing module is connected to the minimum hash processing module.

Images