Method and device for identifying web access portal

Abstract

The embodiments of the present disclosure disclose a method and device for identifying a web access portal. The specific implementation of the method includes: obtaining a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; classifying URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; splitting each initial set into at least one subset based on the decision tree principle, and generating a multi-level word tree based on each initial set and its subset; determining semantic words and parameter words from each multi-level word tree; and outputting the combination of semantic words and parameter words that have been determined in the URL set as a web access portal. This implementation can accurately identify position parameters from a URL data set to determine the actual conscious web access portal, thereby improving the accuracy of security detection and performance monitoring and the convenience of resource management.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method and device for identifying a web access portal. Background Art

[0002] Currently, the main way to access Internet applications is web access based on HTTP (HyperText Transfer Protocol). Security systems and monitoring systems will capture all or a certain percentage of HTTP access traffic for security testing, performance monitoring, etc.

[0003] The web access portal is reflected in the URL (Uniform Resource Locator). The accuracy of the access portal plays a vital role in security systems, monitoring systems, etc.

[0004] The existing technology mainly adopts the manual reporting method to collect the position parameters in the URL path, which has high labor costs. There is also a method of using regular expression matching to identify the position parameters, but the accuracy is low. Summary of the invention

[0005] Embodiments of the present disclosure provide a method and apparatus for identifying a web access portal.

[0006] In the first aspect, an embodiment of the present disclosure provides a method for identifying a web access portal, including: obtaining a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; classifying URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; splitting each initial set into at least one subset based on the decision tree principle, and generating a multi-level word tree based on each initial set and its subset; determining semantic words and parameter words from each multi-level word tree; and outputting the combination of semantic words and parameter words that have been determined in the URL set as a web access portal. Parameter words refer to the location parameters of web access in the usual sense. The determination of parameter words can determine the URL as a web application portal in a practical sense.

[0007] In some embodiments, after obtaining the set of URLs to be identified, the method further includes: for each URL, counting the number of words separated by “ / ” in the path of the URL as the path length of the URL.

[0008] In some embodiments, URLs with the same protocol standard, the same domain name, and the same path length in a URL collection are classified into an initial collection to obtain at least one initial collection, including: creating a hash mapping table, wherein the key of the hash mapping table represents a combination of the protocol standard, the domain name, and the path length, and the value of the hash mapping table represents a list of stored URLs; traversing the URLs in the URL collection, forming a key with the protocol standard, domain name, and path length of the current URL, and taking out the corresponding list from the hash mapping table through the key. If there is no corresponding list, creating a new list and putting the current URL into the list corresponding to the key; determining each key and the corresponding list in the hash mapping table as an initial collection.

[0009] In some embodiments, each initial set is split into at least one subset based on the decision tree principle, and a multi-level word tree is generated based on each initial set and its subset, including: performing the following splitting steps based on the initial set: calculating the position and number of occurrences of each non-classified word in the path of each URL in the initial set; determining the word with the most occurrences in the initial set as a classified word and using it as a node of the multi-level word tree, and splitting the URL corresponding to the determined classified word from the initial set to form a new URL classification set; if the initial set is not empty, repeating the above splitting steps; if the initial set is empty, continuing to perform the above splitting steps with each new URL classification set as the initial set until the splitting cannot be continued.

[0010] In some embodiments, semantic words and parameter words are determined in each multi-level word tree, including: for each multi-level word tree, traversal is started from the top-level node of the multi-level word tree, and when it is found that the number of child nodes of a node is greater than or equal to a predetermined threshold, the parent node of the node until the top-level node is determined as a semantic word, and the nodes below the node are determined as parameter words.

[0011] In some embodiments, the method further includes: verifying the predetermined threshold using a test set of known web access portals of the same order of magnitude as the URL set; if the number of verified classification words exceeds an expected value, increasing the predetermined threshold.

[0012] In some embodiments, the method further includes: performing security detection and / or performance monitoring of network access traffic based on the web access portal.

[0013] In a second aspect, an embodiment of the present disclosure provides a device for identifying a web access portal, comprising: an acquisition unit, configured to acquire a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; a classification unit, configured to classify URLs in the URL set with the same protocol standard, the same domain name, and the same path length into an initial set, and obtain at least one initial set; a splitting unit, configured to split each initial set into at least one subset based on the decision tree principle, and generate a multi-level word tree based on each initial set and its subset; a determination unit, configured to determine semantic words and parameter words from each multi-level word tree; and an output unit, configured to output a combination of semantic words and parameter words that have been determined in the URL set as a web access portal.

[0014] In some embodiments, the device further comprises a statistical unit configured to: after acquiring the set of URLs to be identified, for each URL, count the number of words separated by “ / ” in the path of the URL as the path length of the URL.

[0015] In some embodiments, the classification unit is further configured to: create a hash mapping table, wherein the key of the hash mapping table represents a combination of a protocol standard, a domain name, and a path length, and the value of the hash mapping table represents a list of stored URLs; traverse the URLs in the URL set, form a key with the protocol standard, domain name, and path length of the current URL, and retrieve the corresponding list from the hash mapping table through the key. If there is no corresponding list, create a new list and put the current URL into the list corresponding to the key; determine each key and the corresponding list in the hash mapping table as an initial set.

[0016] In some embodiments, the splitting unit is further configured to: perform the following splitting steps based on the initial set: calculate the position and number of occurrences of each non-classified word in the path of each URL in the initial set; determine the word that appears most times in the initial set as a classified word and use it as a node of a multi-level word tree, and split the URL corresponding to the determined classified word from the initial set to form a new URL classification set; if the initial set is not empty, repeat the above splitting steps; if the initial set is empty, continue to perform the above splitting steps with each new URL classification set as the initial set until the splitting cannot be continued.

[0017] In some embodiments, the splitting unit is further configured to: for each multi-level word tree, start traversing from the top-level node of the multi-level word tree, and when it is found that the number of child nodes of a node is greater than or equal to a predetermined threshold, the parent node of the node until the top-level node is determined as a semantic word, and the nodes below the node are determined as parameter words.

[0018] In some embodiments, the device further comprises an adjustment unit configured to: verify the predetermined threshold using a test set of known web access portals of the same order of magnitude as the URL set; and increase the predetermined threshold if the number of verified classification words exceeds an expected value.

[0019] In some embodiments, the device also includes an application unit configured to: perform security detection and / or performance monitoring of network access traffic based on the web access portal.

[0020] In a third aspect, an embodiment of the present disclosure provides an electronic device for identifying web access portals, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the first aspects.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of the first aspects is implemented.

[0022] The method and device for identifying web access portals provided by the embodiments of the present disclosure can effectively identify which positions in the data of the actual path are position parameters from the data of the actual HTTP access traffic itself based on an iterative data analysis algorithm. Once the web portal can be accurately identified, the web portals in the asset system can be effective and centralized, the network protection system (such as WAF (Web Application Firewall, website application-level intrusion prevention system) can be targeted and finely controlled, and the distributed monitoring system can enable efficient statistics and analysis of problems based on the web portal. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0025] Figure 2 is a flow chart of an embodiment of a method for identifying a web access portal according to the present disclosure;

[0026] Figure 3 is a flow chart of a URL set classification process in a method for identifying a web access portal according to the present disclosure;

[0027] Figure 4a-4bis a flow chart of a URL set splitting process in a method for identifying a web access portal according to the present disclosure;

[0028] Figure 5 is a schematic diagram of a multi-level word tree of a method for identifying a web access portal according to the present disclosure;

[0029] Figure 6 is a structural diagram of an embodiment of a device for identifying a web access portal according to the present disclosure;

[0030] Figure 7 It is a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for identifying a web access portal or an apparatus for identifying a web access portal of the present disclosure can be applied.

[0034] like Figure 1 As shown, system architecture 100 may include websites 101, 102, 103, network 104 and server 105. Network 104 is used to provide a medium for communication links between websites 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0035] Users can use server 105 to interact with websites 101, 102, and 103 through network 104 to receive or send messages, etc. Websites 101, 102, and 103 may be websites that provide various network application services, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] The server 105 may be a server providing URL analysis services, such as a background analysis server providing analysis support for http requests for accessing websites 101, 102, and 103. The background analysis server may analyze and process data such as http requests received by the website, and feed back the processing results (such as web access portals) to the website, and may also set up security systems and monitoring systems for each website based on the web access portals.

[0037] The URL of a web application consists of three important parts:

[0038]

[0039] Through the information in the http protocol in the network access traffic, it is easy to extract the above three parts to form an access entrance.

[0040] However, the information in the Path not only contains the function entry point information, but also contains parameters, such as " / user / 12345 / detail" in the above example, where "user" and "detail" are function entries representing user detailed information, but the "12345" in the middle represents the current user ID. This type of parameter is called a positional parameter.

[0041] Due to the emergence of position parameters, the same functional entry point will generate a large number of similar paths in the traffic but with different position parameters. If these position parameters cannot be accurately identified, subsequent "interface asset management", "performance monitoring", "security testing" and other tasks will either not correspond to the actual entry of the application, and differentiation strategies cannot be adopted, or data will be flooded, resulting in the failure of the entire system.

[0042] Accurately identifying the web access portal means being able to correctly identify the URL path with positional parameters as the same portal.

[0043] For example, the following three paths are found in the traffic:

[0044]

[0045] Correct identification should be able to identify " / api / user / u12345 / info" and " / api / user / u23456 / info" as the same entry " / api / user / ${} / info"

[0046] If it cannot be identified, it is conceivable that the impact on subsequent processing will be huge.

[0047] 1. The web portal in the asset system will cause meaningless expansion of data, and the storage capacity is huge and cannot be used.

[0048] 2. The access portal cannot perform fine control over network protection systems (such as WAF), etc. This will have a huge impact on large Internet companies with huge external network access volume.

[0049] 3. Distributed monitoring systems will be significantly affected. Imagine that there are tens of thousands of entry lists with parameters. Not only is it difficult to count, but it will also overwhelm the calls to the problematic entry.

[0050] In general, the value of a parameter may be a combination of letters and numbers, and there is no strict distinction between non-parameter words. There is no regularity in the position of the parameter, and there may be multiple positional parameters in a path. Even for the same function entry, positional parameters may appear in different positions due to different habits and design principles of developers. See the following example with positional parameters:

[0051]

[0052] It is difficult to make a general regular expression to identify positional parameters just by observing the "actual path" data.

[0053] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0054] It should be noted that the method for identifying a web access portal provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for identifying a web access portal is generally disposed in the server 105 .

[0055] It should be understood that Figure 1 The number of websites, networks and servers in the embodiment is only for illustration. Any number of websites, networks and servers may be provided as required.

[0056] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for identifying a web access portal according to the present disclosure. The method for identifying a web access portal comprises the following steps:

[0057] Step 201, obtaining a set of URLs to be identified.

[0058] In this embodiment, the execution subject of the method for identifying a web access portal (eg Figure 1 The server shown in FIG. 1 can obtain a set of URLs to be identified from various websites, wherein each URL includes a protocol standard, a domain name, and a path.

[0059] For the convenience of description, the following terms are defined first:

[0060] Path: The Path part of the HTTP request. In Web services, it is usually used as the function entry point, also known as the endpoint.

[0061] For example, " / api / user / info / u12345"

[0062] Word: The part of a Path separated by " / ". For example, the words in the Path " / api / user / info / u12345" are "api", "user", "info", "u12345". Furthermore, a Path can be represented as a word list.

[0063] ·Semantic Word: refers to the non-parameter part of the path, such as "api", "user", and "info" in the path " / api / user / info / u12345".

[0064] Parameter Word: refers to the word that is part of the parameter in the path, such as "u12345" in the path " / api / user / info / u12345".

[0065] Path Pattern: refers to setting the positional parameters in the Path to the wildcard ${} format. The Path Pattern of the Path " / api / user / info / u12345" is " / api / user / info / ${}". ${} is a wildcard for any word. It can be seen that the Path Pattern is the real web entrance. If there is no parameter word in a Path, the Path itself can be regarded as a Path Pattern.

[0066] Path Length: refers to the number of words in the path. For example, the length of the path " / api / user / info / u12345" is 4.

[0067] From the above definition, it can be concluded that accurate identification of web portals can be converted into the following description:

[0068] Given a set of Path data, find all its Path Patterns.

[0069] Step 202: classify the URLs in the URL set with the same protocol standard, the same domain name, and the same path length into an initial set to obtain at least one initial set.

[0070] In this embodiment, the path lengths of some websites are known and do not need to be calculated. If the path length is unknown, for each URL, the number of words separated by “ / ” in the path of the URL is counted as the path length of the URL.

[0071] You can find a subset of URLs with the same protocol standard and domain name through string matching, and then divide the subset of URLs into different initial sets according to the path length. You can also first find a subset of URLs with the same path length, and then find the URLs with the same protocol standard and domain name through string matching to form an initial set.

[0072] In some optional implementations of this embodiment, such as Figure 3 As shown, URLs with the same protocol standard, the same domain name, and the same path length in the URL set are classified into an initial set, and at least one initial set is obtained, including:

[0073] Step 301: Create a hash mapping table.

[0074] In this embodiment, the key of the hash mapping table represents a combination of the protocol standard, the domain name, and the path length, and the value of the hash mapping table represents a list of stored URLs. Initially, the hash mapping table is an empty table.

[0075] Step 302, traverse the URLs in the URL set.

[0076] In this embodiment, an unclassified URL is taken out from the URL set each time as the current URL.

[0077] Step 303, the protocol standard, domain name, and path length of the current URL are combined into a key.

[0078] In this embodiment, the path length of the URL can be obtained in advance, or the number of words separated by “ / ” in the path of the URL can be counted as the path length of the URL.

[0079] Step 304: retrieve the corresponding list from the hash map by key; if there is no corresponding list, create a new list.

[0080] In this embodiment, if the protocol standard, domain name, and path length of the current URL appear for the first time, the hash map does not have a corresponding list, and a new one needs to be created. If the key does not appear for the first time, the list corresponding to the key can be found.

[0081] Step 305, put the current URL into the list corresponding to the key.

[0082] In this embodiment, no matter whether the key appears for the first time or has appeared before, the current URL needs to be stored in the list corresponding to the key.

[0083] Step 306: determine each key and the corresponding list in the hash mapping table as an initial set.

[0084] In this embodiment, if the URL traversal is completed, the content filling of the hash mapping table is completed. Each key and the corresponding list in the hash mapping table are determined as an initial set.

[0085] Step 203: split each initial set into at least one subset based on the decision tree principle, and generate a multi-level word tree based on each initial set and its subset.

[0086] In this embodiment, in this step, since the schema, hostname and pathlength in the set URLs are the same, the problem is converted into classifying the Paths in the set into Path Patterns.

[0087] The key to solving the problem is to identify the parameter word in the Path.

[0088] First, we describe the process of the recognition algorithm. For the convenience of description, we first define the definitions of all the data structures used:

[0089] URL category set SET-N <Word-L1,Word-L2,…,Word-L N >: This set indicates that all URLPaths in the set have N words in common, and the positions of these N words in the Path are L1, L2, ... L N For example, the following collection:

[0090]

[0091] N is called the level of this classification set.

[0092] URL category sub-branch: If the category words of one category set contain the category words of another category set and the order is the same, the latter is called the parent node of the former, and the former is called the sub-branch of the latter. If the level of the sub-branch is one more than the level of the parent node, it is called the direct sub-branch of the parent node.

[0093] Classification word: a URL classification set SET-N <Word-L1,Word-L2,…,Word-L N >, Word-L1, Word-L2, these words that all URLs in this set have are called the classification words of this set.

[0094] Number of word occurrences: In a URL classification set SET, the number of times the same word Word appears in a certain position L1 in the path of all URLs is recorded as Count(Word, L1). For example, in the following set,

[0095]

[0096] Count("api",1)=4, count("grp",2)=2, count("grp",1)=1.

[0097] You can search for different words in the same initial set from left to right to split them, and when you encounter different words, you can divide them into different URL category sets. This method can split quickly.

[0098] In some optional implementations of this embodiment, the following splitting steps are performed based on the initial set: the position and number of occurrences of each non-classified word in the path of each URL in the initial set are calculated; the word with the most occurrences in the initial set is determined as a classified word and used as a node of a multi-level word tree, and the URL corresponding to the determined classified word is split from the initial set to form a new URL classification set; if the initial set is not empty, the above splitting steps are repeated;

[0099] If the initial set is empty, each new URL category set is used as the initial set to continue the above splitting steps until further splitting is impossible.

[0100] The instructions and flow chart for constructing a multi-level tree structure are shown in 4a-4b:

[0101] 1. It is known that the Path Length of the current URL category set is N

[0102] 2. Repeat the following process from 1 to N

[0103] 2.1 Assume that the current cycle is in round K

[0104] 2.2 Calculate the number of occurrences of all non-"classification words" in the current "URL classification set". Note that the number of occurrences requires that both the word itself and the word position are the same. When the "URL classification set" is cycled for the first round, it means calculating the number of occurrences of all words.

[0105] 2.3 Find the most frequent words (including positions) "Word-L K+1 , L K+1 ”

[0106] 2.4 "Word-L K+1 , L K+1 " is taken out from the current classification set SET-K to form a new URL classification set. The first K classification words of the new URL classification set obviously come from the K classification words (including the order) of the current set. The new URL classification set is a direct child branch of the current classification set.

[0107] 2.5 If there is no URL element in the current category set, it means that the current URL category set has been completely split into multiple direct sub-branch category sets. If there is still a URL element, go back to 2.2 and start counting the number and splitting again.

[0108] 3. Once all the current K-level K-classification subsets have been split, multiple K+1-level classification subsets will be obtained. Then the K+1 round cycle will be entered to continue splitting all the newly generated K+1-level classification sets. Finally, multiple N-level classification subsets will be obtained.

[0109] 4. At this point, all the classification words of the N-level classification set can actually form a word tree with a depth of N. The classification word with K=1 is the top layer of the tree, and the words of its direct child branches are the child nodes of the corresponding tree.

[0110] Algorithm implementation example: Take the following set as an example

[0111]

[0112] Obviously, this is a classification set with Path Length=4.

[0113] 1. K = 1 Count the number of occurrences of all words. count("api", 1) = 4, count("user", 2) = 2, count("grp", 1) = 1, count("detail", 4) = 4, and the rest are omitted. Obviously, the maximum number is count("api", 1) = 4 and count("detail", 4) = 4.

[0114] 2. Take count("api",1)=4 as the maximum number, then split the first-level subset SET-1<"api"-1>={ / api / user / u1234 / detail, / api / user / u2345 / detail, / api / grp / g1234 / detail, / api / grp / g34 / detail}.

[0115] 3. The original set becomes SET-0<>=“ / grp / g123 / user / u4567”.

[0116] 4.K=1 completed.

[0117] 5. Starting with K=2, for SET-1<"api"-1>={ / api / user / u1234 / detail, / api / user / u2345 / detail, / api / grp / g1234 / detail, / api / grp / g34 / detail}. Calculate the number of word occurrences, count("user",2)=2, count("grp",1)=1, count("detail",4)=4, the maximum number is count("detail",4)=4

[0118] 6. Take count("detail",4)=4 as the maximum value and split it into SET-2<"api"-1,"detail"-4>={ / api / user / u1234 / detail, / api / user / u2345 / detail, / api / grp / g1234 / detail, / api / grp / g34 / detail}.

[0119] 7. Omit the intermediate process and finally get the following 4-level set

[0120] SET-4<"api"-1,"detail"-4,"user"-2,"u1234"-3>

[0121] SET-4<"api"-1,"detail"-4,"user"-2,"u2345"-3>

[0122] SET-4<"api"-1,"detail"-4,"grp"-2,"g1234"-3>

[0123] SET-4<"api"-1,"detail"-4,"grp"-2,"g34"-3>

[0124] SET-4<"grp"-1,"g123"-2,"user"-3,"u4567"-4>

[0125] 8 Thus, a multi-level word tree structure is obtained, such as Figure 5 shown.

[0126] Step 204, determining semantic words and parameter words from each multi-level word tree.

[0127] In this embodiment, for each multi-level word tree, traversal is performed starting from the top-level node of the multi-level word tree, and words corresponding to nodes of a predetermined depth are selected as semantic words, and words of other nodes are selected as parameter words.

[0128] In some optional implementations of the present embodiment, for each multi-level word tree, traversal is performed starting from the top-level node of the multi-level word tree. When it is found that the number of child nodes of a node is greater than or equal to a predetermined threshold, the parent node of the node up to the top-level node is determined as a semantic word, and the nodes below the node are determined as parameter words.

[0129] The meaning of the predetermined threshold T is to distinguish the number of sub-categories with semantics generated by human design from the number of sub-categories generated by parameter changes. In WebAPI design, all direct sub-methods belonging to a certain category generally do not exceed 5-20. If there are too many, it is not conducive to the use and communication of developers. Generally, developers will aggregate more sub-methods again as secondary sub-methods.

[0130] For example, there are many sub-methods about user, but they are divided into two levels of classification, such as information query methods and modification methods, so the following API will appear

[0131] / api / user / query / info

[0132] / api / user / query / org

[0133] / api / user / create /

[0134] / api / user / update /

[0135] However, due to the positional parameters, there are a certain number of sub-branches, even in a small scale, there will be a large number of branches, such as: (where position 3 is the userID positional parameter)

[0136] / api / user / u1234 / query / info

[0137] / api / user / u2344 / query / info

[0138] …

[0139] / api / user / u8765 / query / info

[0140] Therefore, a T value between 5 and 20 is generally more appropriate.

[0141] In this document, due to space limitations and a small amount of data, T=2.

[0142] Starting from the top node of the word tree, traverse, when it is found that the number of child branches of a node is greater than or equal to T, the parent node of this node (including this node) until the top node is a semantic word, and the nodes below this node (child nodes and grandchild nodes, etc.) are all parameter words.

[0143] In some optional implementations of this embodiment, the method further includes: verifying the predetermined threshold using a test set of known web access portals of the same order of magnitude as the URL set; if the number of verified classification words exceeds the expected value, the predetermined threshold is increased. The value of T affects the accuracy of recognition, and empirical values ​​can be used for recognition, and then the obtained recognition results are analyzed. If T is small, the location words will be misjudged as classification words. Developers can use test sets of known location words and classification words to analyze whether there are any misjudgments. If a misjudgment occurs, adjust the T value. Similarly, if a classification word is misjudged as a location word, the T value needs to be reduced.

[0144] Step 205: output the combination of the semantic word and the parameter word that has been determined in the URL set as a web access entry.

[0145] In this embodiment, the Path Pattern is the web entry. As shown in the above example, three PathPattern can be obtained:

[0146] {"api"-1, "detail"-4, "user"-2, ${}-3}, the URL's Path Pattern is / api / user / ${} / detail

[0147] {"api"-1,"detail"-4,"grpr"-2,${}-3}, URL Path Pattern is / api / grp / ${} / detail

[0148] {"grp"-1,"g123"-2,"user"-3,"u4567"-4}, URL Path Pattern is / grp / g123 / user / u4567.

[0149] Obviously, the first two results are very accurate, while the third result is obviously not correct due to the small amount of data. However, it is obvious that the third result will be significantly improved with only a small increase in the amount of data.

[0150] In actual work, due to the small amount of data, some low-quality results will be produced. Just accumulating this data and then re-applying this process will produce obvious results.

[0151] In some optional implementations of this embodiment, the method further includes: performing security detection and / or performance monitoring of network access traffic according to the web access portal. Protection can be accurately enabled to minimize the impact and improve defense capabilities. For example, rules to prevent SQL injection can be set according to the web portal. The number of calls to the web portal, the average response time, etc. can also be counted. Problems with the website can be quickly located.

[0152] From the above algorithm process, we can see that there are three key points in the algorithm:

[0153] 1. "Semantic words will definitely have a higher number of occurrences than classifiers", so the semantic words are first found out based on the number of occurrences of the words as the top and leading nodes of the classifiers. The following is the proof:

[0154] Given the current path set, where the number of "parameter words" at the same position in all possible Path Patterns is greater than 1, if there is a word at a position that is a semantic word, then the word with the highest number of occurrences is a "semantic word" or all the words in the path are "parameter words".

[0155] Proof by contradiction:

[0156] Assume that the word with the highest number of occurrences is WordA, and the number of occurrences is NA. Assume that WordA is the parameter word.

[0157] Let WordA appear in PathA,

[0158] There are two cases: PathA contains semantic words and PathA does not contain semantic words.

[0159] The first case: PathA contains semantic words. Suppose there is a semantic word WordSEC in it. According to the definition, if WordA appears, WordSEC will definitely appear in the same Path. Therefore, the number of occurrences of WordSEC is not less than the number of occurrences of WordA. Considering that the number of "parameter words" in the same position in all possible Path Patterns is greater than 1, that is, there is WordB, which is also a parameter word of the same Pattern as PathA, then the appearance of WordB will also cause the appearance of WordSEC. So far, Count(WordSEC)>=count(WordA)+count(WordB)>count(wordA), which contradicts the original assumption that wordA has the highest number of occurrences.

[0160] The second case: PathA does not contain any semantic words, which obviously conforms to the premise that all the words in Path are "parameter words".

[0161] Therefore, unless there are no semantic words, "semantic words will definitely appear more frequently than classifiers"

[0162] 2. Using the basic principle of decision tree, the data set is divided into various subsets after judgment, and decision is made again (step 203), thus avoiding data interference between different subsets.

[0163] For example, the actual pattern of the following collection is { / api / grp / ${} / user / ${} / update, / api / create / grp / ${}

[0164]

[0165] If we simply count the number of word occurrences, we will find that the parameter word "g133"-1 appears more often than the semantic word "create"-2, which will cause g133 to be mistakenly regarded as a semantic word.

[0166] However, by dividing into subsets after the "decision" in step 203, this interference is well avoided.

[0167] 3. When humans design APIs, due to the limited human processing capabilities, semantic words are naturally classified. This is obviously different from the large number of branches caused by parameter words, and empirical values ​​can be designed to distinguish them in turn. See the process of determining the threshold T for splitting in step 204.

[0168] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for identifying a web access portal. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0169] like Figure 6 As shown, the apparatus 600 for identifying a web access portal in this embodiment includes: an acquisition unit 601, a classification unit 602, a splitting unit 603, a determination unit 604, and an output unit 605. The acquisition unit 601 is configured to acquire a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; the classification unit 602 is configured to classify URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; the splitting unit 603 is configured to split each initial set into at least one subset based on the decision tree principle, and generate a multi-level word tree based on each initial set and its subset; the determination unit 604 is configured to determine semantic words and parameter words from each multi-level word tree; and the output unit 605 is configured to output the combination of semantic words and parameter words that have been determined in the URL set as a web access portal.

[0170] In this embodiment, the specific processing of the acquisition unit 601, the classification unit 602, the splitting unit 603, the determination unit 604, and the output unit 605 of the device 600 for identifying a web access portal can be referred to. Figure 2 Corresponding to step 201, step 202, step 203, step 204 and step 205 in the embodiment.

[0171] In some optional implementations of this embodiment, the device also includes a statistical unit (not shown in the drawings), which is configured to: after obtaining the set of URLs to be identified, for each URL, count the number of words separated by “ / ” in the path of the URL as the path length of the URL.

[0172] In some optional implementations of this embodiment, the classification unit 602 is further configured to: create a hash mapping table, wherein the key of the hash mapping table represents a combination of the protocol standard, domain name, and path length, and the value of the hash mapping table represents a list of stored URLs; traverse the URLs in the URL set, form a key with the protocol standard, domain name, and path length of the current URL, and retrieve the corresponding list from the hash mapping table through the key. If there is no corresponding list, create a new list and put the current URL into the list corresponding to the key; determine each key and the corresponding list in the hash mapping table as an initial set.

[0173] In some optional implementations of this embodiment, the splitting unit 603 is further configured to: perform the following splitting steps based on the initial set: calculate the position and number of occurrences of each non-classified word in the path of each URL in the initial set; determine the word that appears most times in the initial set as a classified word and use it as a node of the multi-level word tree, and split the URL corresponding to the determined classified word from the initial set to form a new URL classification set; if the initial set is not empty, repeat the above splitting steps; if the initial set is empty, continue to perform the above splitting steps with each new URL classification set as the initial set until the splitting cannot be continued.

[0174] In some optional implementations of the present embodiment, the splitting unit 603 is further configured to: for each multi-level word tree, traverse starting from the top-level node of the multi-level word tree, and when it is found that the number of child nodes of a node is greater than or equal to a predetermined threshold, the parent node of the node until the top-level node is determined as a semantic word, and the nodes below the node are determined as parameter words.

[0175] In some optional implementations of this embodiment, the device also includes an adjustment unit (not shown in the drawings), which is configured to: verify the predetermined threshold using a test set of known web access portals of the same order of magnitude as the URL set; if the number of verified classification words exceeds the expected value, increase the predetermined threshold.

[0176] In some optional implementations of this embodiment, the device further includes an application unit (not shown in the drawings) configured to: perform security detection and / or performance monitoring of network access traffic according to the web access portal.

[0177] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of a server or website in FIG. 700. Figure 7 The website / server shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0178] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0179] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0180] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0181] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; classifies the URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; splits each initial set into at least one subset based on the decision tree principle, and generates a multi-level word tree based on each initial set and its subset; determines semantic words and parameter words from each multi-level word tree; and outputs the combination of semantic words and parameter words that have been determined in the URL set as a web access entry.

[0182] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a classification unit, a splitting unit, a determination unit, and an output unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the acquisition unit may also be described as a "unit for acquiring a set of URLs to be identified".

[0185] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A method for identifying a web access portal, comprising: Obtain a set of URLs to be identified, where each URL includes a protocol standard, a domain name, and a path; Classifying URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; Based on the decision tree principle, each initial set is split into at least one subset, and a multi-level word tree is generated based on each initial set and its subset; Determine semantic words and parameter words from each multi-level word tree; Outputting the combination of the semantic word and the parameter word determined in the URL set as a web access entry; The method of splitting each initial set into at least one subset based on the decision tree principle and generating a multi-level word tree based on each initial set and its subsets includes: The following splitting steps are performed based on the initial set: the position and number of occurrences of each non-classified word in the path of each URL in the initial set are calculated; the word with the most occurrences in the initial set is determined as a classified word and used as a node of a multi-level word tree, and the URL corresponding to the determined classified word is split from the initial set to form a new URL classification set; if the initial set is not empty, the above splitting steps are repeated; If the initial set is empty, each new URL category set is used as the initial set to continue the above splitting steps until further splitting is impossible.

2. The method according to claim 1, wherein: After obtaining the URL set to be identified, the method further includes: For each URL, the number of words separated by " / " in the path of the URL is counted as the path length of the URL.

3. The method according to claim 1, wherein: The URLs in the URL set with the same protocol standard, the same domain name, and the same path length are classified into an initial set to obtain at least one initial set, including: Creating a hash mapping table, wherein the key of the hash mapping table represents a combination of a protocol standard, a domain name, and a path length, and the value of the hash mapping table represents a list of stored URLs; Traverse the URLs in the URL set, form a key with the protocol standard, domain name, and path length of the current URL, and retrieve the corresponding list from the hash map by the key. If there is no corresponding list, create a new list and put the current URL into the list corresponding to the key; Each key and the corresponding list in the hash mapping table is determined as an initial set.

4. The method according to claim 1, wherein: Determining semantic words and parameter words from each multi-level word tree includes: For each multi-level word tree, traversal is started from the top-level node of the multi-level word tree. When it is found that the number of child nodes of a node is greater than or equal to a predetermined threshold, the parent node of the node until the top-level node is determined as a semantic word, and the nodes below the node are determined as parameter words.

5. The method according to claim 4, wherein: The method further comprises: Verifying the predetermined threshold using a test set of known web access portals of the same order of magnitude as the URL set; If the number of verified classified words exceeds the expected value, the predetermined threshold is increased.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Security detection and / or performance monitoring of network access traffic is performed based on the web access portal.

7. A device for identifying a web access portal, comprising: An acquisition unit is configured to acquire a set of URLs to be identified, wherein each URL includes a protocol standard, a domain name, and a path; A classification unit is configured to classify URLs with the same protocol standard, the same domain name, and the same path length in the URL set into an initial set to obtain at least one initial set; A splitting unit, configured to split each initial set into at least one subset based on a decision tree principle, and generate a multi-level word tree based on each initial set and its subset; a determination unit configured to determine semantic words and parameter words from each multi-level word tree; an output unit configured to output the combination of the semantic word and the parameter word determined in the URL set as a web access entry; The method of splitting each initial set into at least one subset based on the decision tree principle and generating a multi-level word tree based on each initial set and its subsets includes: The following splitting steps are performed based on the initial set: the position and number of occurrences of each non-classified word in the path of each URL in the initial set are calculated; the word with the most occurrences in the initial set is determined as a classified word and used as a node of a multi-level word tree, and the URL corresponding to the determined classified word is split from the initial set to form a new URL classification set; if the initial set is not empty, the above splitting steps are repeated; If the initial set is empty, each new URL category set is used as the initial set to continue the above splitting steps until further splitting is impossible.

8. An electronic device for identifying a web access portal, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images