Method, device, computer equipment and storage medium for classifying website

Abstract

The present application relates to a method, device, computer equipment and storage medium for classifying web addresses. The method comprises: obtaining a web address to be classified, performing field segmentation on the web address to be classified to obtain a target field corresponding to a first node relationship; obtaining a target clustering structure tree; matching each target field with each node in the target clustering structure tree layer by layer according to the first node relationship; when each target field matches each node in a target path, determining the web address category of the web address to be classified according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree. The above scheme can directly implement parallel computing based on the target clustering structure tree to classify the web addresses to be classified, quickly output the web address classification results of the web addresses to be classified, and effectively ensure the efficiency of web address classification.

Description

Technical Field

[0001] The present application relates to the field of network technology, and in particular to a method, device, computer equipment and storage medium for classifying URLs. Background Art

[0002] In order to continuously optimize the speed and experience of Web (World Wide Web) applications, users' HTTP (Hyper Text Transfer Protocol) requests are generally collected and analyzed to better identify the shortcomings in the application and then perform targeted optimization.

[0003] Based on this, URLs (Uniform Resource Locator) and other websites need to be classified. In traditional technology, URL clustering is achieved through K-means algorithm (k-means clustering algorithm). Generally, the URL is converted into a matrix for similarity calculation, and the URL clustering is achieved based on the similarity.

[0004] However, if the number of URLs is huge, traditional technologies will cause large performance losses during the conversion process, resulting in inefficient URL classification.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] Based on this, it is necessary to provide a URL classification method, device, computer equipment and storage medium that can improve the efficiency of URL classification in response to the above technical problems.

[0007] A method for classifying web addresses, the method comprising:

[0008] Acquire a website to be classified, and perform field segmentation on the website to be classified to obtain a target field corresponding to the first node relationship;

[0009] Obtain a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs;

[0010] Matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship;

[0011] When each of the target fields matches each of the nodes in the target path, the URL category of the URL to be classified is determined according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0012] A device for classifying web addresses, the device comprising:

[0013] A module for obtaining a website address to be classified, used for obtaining a website address to be classified, and performing field segmentation on the website address to be classified to obtain a target field corresponding to the first node relationship;

[0014] A structure tree acquisition module is used to acquire a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the conditions, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs;

[0015] A layer-by-layer matching module, used for matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship;

[0016] The website category determination module is used to determine the website category of the website to be classified according to the target path when each of the target fields matches each node in the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Acquire a website to be classified, and perform field segmentation on the website to be classified to obtain a target field corresponding to the first node relationship;

[0019] Obtain a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs;

[0020] Matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship;

[0021] When each of the target fields matches each of the nodes in the target path, the URL category of the URL to be classified is determined according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] Acquire a website to be classified, and perform field segmentation on the website to be classified to obtain a target field corresponding to the first node relationship;

[0024] Obtain a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs;

[0025] Matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship;

[0026] When each of the target fields matches each of the nodes in the target path, the URL category of the URL to be classified is determined according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0027] The above-mentioned URL classification method, device, computer equipment and storage medium obtain the target field corresponding to the first node relationship after field segmentation of the URL to be classified; obtain the target clustering structure tree; match each target field with each node in the target clustering structure tree layer by layer according to the first node relationship; when each target field matches the target path from the root node to the leaf node in the target clustering structure tree, determine the URL category of the URL to be classified according to the target path. The above-mentioned technical solution can directly classify the URL to be classified based on the pre-constructed target clustering structure tree, quickly output the URL classification result of the URL to be classified, and effectively ensure the efficiency of URL classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A diagram of an application environment of a method for classifying URLs in an embodiment;

[0029] Figure 2 A schematic diagram of a flow chart of a method for classifying URLs in one embodiment;

[0030] Figure 3 A schematic diagram of a process of URL clustering in one embodiment;

[0031] Figure 4 A schematic diagram of a process of URL clustering in another embodiment;

[0032] Figure 5 A schematic diagram of a process of URL clustering in yet another embodiment;

[0033] Figure 6 A schematic diagram of a process of URL clustering in another embodiment;

[0034] Figure 7A schematic diagram of a process of obtaining a URL by a server in one embodiment;

[0035] Figure 8 A schematic diagram of a process of URL clustering in another embodiment;

[0036] Fig. 9 A schematic diagram of a window for obtaining website search information in one embodiment;

[0037] Fig.10 A schematic diagram of an interface showing a URL response status analysis result in one embodiment;

[0038] Fig.11 A schematic diagram of an interface showing a result of analyzing a website response status in another embodiment;

[0039] Fig.12 is a structural block diagram of a device for classifying URLs in one embodiment;

[0040] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The classification method, device, computer equipment and storage medium of the website provided in this application can be implemented based on cloud technology. Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0043] The various embodiments of the present application can be further implemented based on cloud computing. Among them, cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0044] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0045] The classification method of URLs provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 101 communicates with the server 102 through the network. The server 102 receives the URL to be classified sent by the terminal 101, and classifies the URL to be classified based on the pre-built target clustering structure tree, and outputs the URL category of the URL to be classified. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0046] In one embodiment, Figure 2 As shown, a method for classifying web addresses is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The method includes the following steps:

[0047] S201, obtaining a website to be classified, and performing field segmentation on the website to be classified to obtain a target field corresponding to a first node relationship.

[0048] Wherein, the web address is also called the web page address, which refers to the address of the web page on the Internet. In one embodiment, the web address may refer to a uniform resource locator (URL, Uniform Resource Locator, i.e., uniform resource locator, location address, URL address). The uniform resource locator is the address of a standard resource on the Internet, like a house number on the network. It was originally invented by Tim Berners-Lee to be used as the address of the World Wide Web, and now it has been compiled by the World Wide Web Consortium as the Internet standard RFC 1738.

[0049] For URL categories, the string formed by aggregating multiple URLs together can be called a clustered URL, and a clustered URL can be used as a URL category. A URL category can correspond to at least one URL that matches it. When you need to analyze a URL of a certain category, you can find the corresponding matching URLs through the URL category, and perform responsiveness analysis on these matching URLs, which can obtain the responsiveness analysis results of the category.

[0050] The URL to be classified may refer to a URL that has not been classified. The URL to be classified may be a URL extracted based on a request to access a certain application platform in a historical time period or at the current moment. The number of URLs to be classified may be one or more. When there are more than one URLs to be classified, these URLs to be classified may be classified synchronously, and the classification results of these URLs to be classified may be output together. The application platform may refer to a platform that can realize specific application functions, such as a video playback platform.

[0051] In one embodiment, the structure of the URL to be classified may be as follows:

[0052] [Protocol type]: / / [Server address]:[Port number] / [Resource level UNIX file path][File name]? [Query]#[Fragment ID]

[0053] Wherein, [Protocol Type] may correspond to a protocol conforming to the above structure, such as HTTP protocol. [Query] is optional and is used to pass parameters to the dynamic web page. The passed parameters may be multiple, and these parameters are separated by "&" symbols, and the name and value of each parameter are separated by "=".

[0054] In one embodiment, the step of obtaining the target field of the corresponding node relationship after segmenting the to-be-classified website into fields comprises: obtaining a segmentation identifier; segmenting the to-be-classified website into at least one field using the segmentation identifier as a separator to obtain the target field. That is, the to-be-classified website is segmented into at least one string using a separator, and each string is used as a target field. When there are more than one to-be-classified website, the to-be-classified website can be segmented into fields respectively to obtain the target fields corresponding to each to-be-classified website.

[0055] The segmentation identifier may be " / ", "&", and "?". The target fields obtained may form a target field list separated by commas.

[0056] Take https: / / demo.qq.com / api / adddata / 1? key1=abc&key2=def as an example. The target field list after segmentation is: https:, demo.qq.com, api, adddata, 1, key1=abc, key2=def. Among them, "https:" is the first field, that is, the field of the first level, "demo.qq.com" is the second field, that is, the field of the second level, and so on for other fields.

[0057] In computer science, a tree is an abstract data type or a data structure that implements this abstract data type, used to simulate a data set with tree-like structure properties. It is a hierarchical set of n (n>0) finite nodes. It is called a "tree" because it looks like an upside-down tree, that is, its roots are facing upwards and its leaves are facing downwards. A structure tree refers to a tree-based data structure, which can be a binary tree.

[0058] In one embodiment, each target field may have a corresponding relationship with a node in a structure tree, and this corresponding relationship is called a node relationship. For example, according to the position of the field (which may be the front and back sorting position in the URL), the first field corresponds to the root node in the structure tree, the subsequent fields correspond to the child nodes of the existing nodes in turn, and the last field corresponds to the leaf node in the structure tree. Therefore, based on the node relationship, the target field can be mapped to the structure tree, and the matching judgment with the structure tree is performed.

[0059] S202, obtaining a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs.

[0060] The sample website refers to a website used to construct a sample structure tree. In one embodiment, the sample website can be a website extracted based on a request to access a certain application platform within a historical time period.

[0061] The URLs extracted from the access request can be stored in a pre-constructed URL collection. When it is necessary to construct a sample structure tree, a set number of URLs in the URL collection can be determined as sample URLs. In one embodiment, the remaining URLs in the URL collection can be sequentially used as URLs to be classified, and the URLs to be classified can be classified according to the matching relationship between the target field of the URL to be classified and the target cluster structure tree. The process of adding URLs to the URL collection can be: the front-end monitoring system monitors the URLs for a specific application platform in real time, and sends these URLs to the server; the server stores them in the URL collection according to a specific structure.

[0062] As mentioned above, the sample structure tree is a structure tree constructed based on the sample fields. That is, the fields in the sample fields are sequentially used as nodes of the structure tree according to the node relationship, and the data structure constructed by these nodes is the sample structure tree.

[0063] In one embodiment, the sample structure tree includes at least one level, and each level corresponds to a node. In the embodiment of the present invention, the target clustering structure tree is obtained by merging the nodes of the first target level in the sample structure tree. Among them, the first target level can be more than one. Specifically, the sample structure tree can be scanned downward layer by layer from the root node. When a certain level L1 is scanned, the number of nodes M1 in the level L1 is determined. When the number of nodes M1 meets the conditions, the level L1 is determined to be the first target level, and the nodes in the level L1 are merged into a node P, and the subsequent nodes are adjusted: the child nodes of each node in the level L1 are used as the child nodes of the node P. At this time, it can be considered that the node merging of the level L1 is completed; then the next level L2 of the level L1 is scanned, and the node merging is completed in the same way as the level L1, until all levels in the sample structure tree are scanned. It should be noted that in the embodiment of the present invention, the process of merging nodes can be called a pruning process. After the pruning process is completed, it can be considered that a specific number (usually a larger number) of URLs are clustered into another specific number (usually a smaller number) of URLs.

[0064] In one embodiment, the condition that the number of nodes of the first target level needs to meet may be that the number of nodes is greater than a node number threshold. The node number threshold may be a fixed value, for example, 2, 5, 10, etc. The node number threshold may not be a fixed value, but may be positively correlated with the number of sample URLs, that is, it may change as the number of sample URLs changes. Specifically, the product of the number of sample URLs and a specific proportional coefficient is determined as the node number threshold, and when the number of nodes of the first target level is greater than or equal to the product, it is considered that the number of nodes of the first target level meets the condition.

[0065] The target clustering structure tree is a structure tree obtained by performing node clustering processing on the pre-constructed structure tree. Specifically, the structure tree obtained after performing node clustering processing on the sample structure tree is determined as the target clustering structure tree.

[0066] In one embodiment, the process of obtaining the target clustering structure tree can be implemented in at least the following ways: 1. directly determining the pre-built structure tree as the target clustering structure tree; 2. obtaining the target clustering structure tree based on clustering of the sample structure trees after obtaining the URLs to be classified.

[0067] S203: Match each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship.

[0068] In this step, nodes matching each target field are determined in the target clustering structure tree in turn according to the first node relationship and matched layer by layer. Matching can refer to determining whether a certain field is completely consistent with the string corresponding to the node or whether the field is included in the string corresponding to the node. The embodiment of the present invention operates on the original string, and there is no need to construct a matrix to calculate the similarity, and the performance of node matching is extremely high.

[0069] In one embodiment, the field level corresponding to each target field can be determined (the level where the target field is located can be called the field level), and the node of the structure tree level corresponding to the field level (each level in the structure tree can be called the structure tree level) can be determined in the target clustering structure tree, and it is judged whether the corresponding target field matches the node. When it matches, the matching judgment of the next level is performed until the target field of the last field level is reached. It should be noted that the field level and the structure tree level can be collectively referred to as the level. In some embodiments, for the convenience of description, the field level and the structure tree level are not distinguished. It should be understood that the level corresponding to the field refers to the field level, and the level related to the structure tree refers to the structure tree level.

[0070] In one embodiment, if there are multiple URLs to be classified, the target fields corresponding to the URLs to be classified are matched with the target clustering structure tree in order to obtain matching results of the URLs to be classified, and then the URL categories corresponding to the URLs to be classified are obtained based on the matching results.

[0071] S204, when each of the target fields matches each of the nodes in the target path, determining the URL category of the URL to be classified according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0072] When each target field can be matched from the root node of the target clustering structure tree to the leaf node in sequence, the path from the root node to the leaf node is determined as the target path. At this time, it is determined that each target field matches each node in the target path, and then the URL category of the URL to be classified is determined according to the target path.

[0073] In one embodiment, the process of determining the URL category of the URL to be classified according to the target path may be: determining a target cluster URL corresponding to the target path, and determining the target cluster URL as the target cluster URL of the URL to be classified.

[0074] In the above-mentioned URL classification method, the target field corresponding to the first node relationship is obtained after field segmentation of the URL to be classified; the target clustering structure tree is obtained; each target field is matched layer by layer with each node in the target clustering structure tree according to the first node relationship; when each target field matches the target path from the root node to the leaf node in the target clustering structure tree, the URL category of the URL to be classified is determined according to the target path.

[0075] In one embodiment, the target clustering structure tree is a structure tree that is pre-clustered based on the sample structure tree. Traditional clustering methods have the problem of forgetting disaster: clustering algorithms process data in batches, that is, the data required for training is given once, and once new data comes in after the model is trained, all the data needs to be merged and retrained, which results in the new model losing information about the old model trained before, resulting in performance loss. The embodiment of the present invention adopts a progressive classification strategy, directly classifies the URLs to be classified based on the target clustering structure tree, and the historical clustering results are directly involved in the current classification, thereby reducing operations and effectively ensuring the efficiency of URL classification.

[0076] In one embodiment, each of the target fields is matched layer by layer with each of the nodes in the target clustering structure tree according to the first node relationship, including: determining the nodes that match each of the target fields layer by layer in the target clustering structure tree according to the first node relationship to obtain the target nodes; when each of the target nodes forms a path from a root node to a leaf node in the target clustering structure tree, determining that each of the target fields matches each of the nodes in the target path.

[0077] In one embodiment, the layer-by-layer matching process may be: obtaining the first field to be matched in the target field; determining the first structure tree level to be matched corresponding to the level of the first field to be matched in the target clustering structure tree according to the first node relationship; when it is determined that the first field to be matched matches the node in the first structure tree level to be matched, determining the second structure tree level to be matched corresponding to the level of the second field to be matched in the target clustering structure tree according to the first node relationship; the second field to be matched is the next field of the first field to be matched; when the second field to be matched matches the node in the second structure tree level to be matched and the second structure tree level to be matched is the level where the leaf node is located, determining that each of the target fields matches each of the nodes in the target path.

[0078] Specifically, the process of performing the matching judgment of the first level can be: obtaining the first target field, and performing matching judgment on the first target field and the root node in the target clustering structure tree. When the first target field matches the root node, performing the matching judgment of the second level: obtaining the second target field, and performing matching judgment on the second target field and the child node of the root node in the target clustering structure tree. When the second target field matches the child node of the root node in the target clustering structure tree, performing the matching judgment of the third level, and so on, until the target field of the last level is judged. When the matching judgment result of a certain level is mismatch, the matching judgment of the next level can be stopped. For example, when the second target field does not match the child node of the root node, the matching judgment process is terminated. At this time, the URL to be classified can be used as a new URL category, and the indication information of "unsuccessful classification" of the URL to be classified is output.

[0079] The above embodiment can accurately determine the target path that matches all target fields by matching the target fields with the nodes of the target clustering structure tree layer by layer, and then classify the URLs to be classified into the URL categories corresponding to the target paths, thereby obtaining accurate URL classification results.

[0080] In one embodiment, the nodes matching each of the target fields are determined layer by layer in the target clustering structure tree according to the first node relationship, and after the target nodes are obtained, it also includes: when each of the target nodes cannot form a path from the root node to the leaf node in the target clustering structure tree, each of the target fields is superimposed into the target clustering structure tree according to the first node relationship to obtain a structure tree to be merged; when a clustering instruction is obtained, the nodes of the second target level whose number of nodes in the structure tree to be merged meets the condition are merged to obtain a clustering update structure tree; the clustering update structure tree is used to classify the new URL to be classified when the new URL to be classified is obtained.

[0081] As described in the above embodiment, when there is a node path (i.e., target path) in the target clustering structure tree that matches all target fields, the URL category of the URL to be classified is obtained according to the URL category corresponding to the node path. In another case, some target fields cannot match the nodes in the target clustering structure tree. For example, the first d target fields match the nodes in the target clustering structure tree, while the d+1th target field does not match the nodes in the target clustering structure tree. At this time, the matching judgment of the subsequent levels is no longer performed, but each target field is added to the target clustering structure tree to obtain the structure tree to be merged. The structure tree to be merged can have at least one more branch path (i.e., the path formed by the target fields that cannot match the nodes in the target clustering structure tree) than the target clustering structure tree.

[0082] Furthermore, when a clustering instruction is obtained, the structure tree to be merged can be clustered according to the process of obtaining the target clustering structure tree from the sample structure tree. That is, the nodes of the second target level in the structure tree to be merged are merged, and the structure tree obtained after merging the nodes of each level is the clustering update structure tree.

[0083] In one embodiment, the clustering instruction may be a trigger instruction automatically generated by the server when the clustering condition is met. Specifically, the server may generate the clustering instruction under at least one of the following conditions: 1. When the number of URLs corresponding to the nodes that have not been processed by the node merging process in the clustering update structure tree is greater than a preset URL number threshold (which can be determined according to actual conditions, and the embodiment of the present invention does not limit this), it is considered that the clustering condition is met and the clustering instruction is generated; 2. When the time period from the last clustering exceeds the preset time period threshold (which can be determined according to actual conditions, and the embodiment of the present invention does not limit this), it is considered that the clustering condition is met and the clustering instruction is generated; 3. When the number of URLs that have not been clustered in the URL set is greater than the preset URL number threshold, it is considered that the clustering condition is met and the clustering instruction is generated. In some embodiments, the clustering instruction may not be generated by the server. Further, the acquisition process of the clustering instruction may also be: the clustering instruction is sent by the input device connected to the server receiver, for example: the input device is a user terminal, and the user terminal sends the clustering instruction to the server when receiving the user's clustering trigger operation through the interface.

[0084] In one embodiment, when each of the target nodes cannot form a path from the root node to the leaf node in the target clustering structure tree, the URL to be classified can be added to the URL set instead of being added to the target clustering structure tree. When the number of unclustered URLs in the URL set meets the condition, it is considered that the clustering condition is met, and the server generates a clustering instruction.

[0085] In the above embodiment, when the classification of the to-be-classified website fails, the to-be-classified website is recorded and the next clustering is awaited, so that the to-be-classified website can be classified as much as possible and the clustering structure tree can be continuously updated to ensure accurate classification of subsequent to-be-classified websites.

[0086] In one embodiment, before obtaining the target clustering structure tree, it also includes: obtaining a sample URL, performing field segmentation on the sample URL to obtain a sample field corresponding to a second node relationship; constructing a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree; performing node merging operations on the sample structure tree layer by layer, so that when the first target level is accessed, the nodes in the first target level are merged into an arbitrary matching node, and the child nodes of each node in the first target level are merged into the child nodes of the arbitrary matching node; the arbitrary matching node can match with any node; and the sample structure tree after the node merging is used as the target clustering structure tree.

[0087] Among them, any matching node in the target clustering structure tree can be represented by symbols such as "*".

[0088] In one embodiment, when classifying the URLs to be classified according to the target clustering structure tree, when determining whether a target field matches any matching node in the target clustering structure tree as needed, it is directly determined to be a match, and the matching of the next level can be determined at this time.

[0089] In one embodiment, field segmentation of the sample URL may be performed in a similar manner to segmentation of the URL to be classified, that is, the field segmentation of the sample URL to obtain the sample field includes: obtaining a segmentation identifier; segmenting the sample URL into at least one field using the segmentation identifier as a separator to obtain a sample field corresponding to the second node relationship.

[0090] In one embodiment, the construction of a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree includes: configuring each of the sample fields as a node in the sample structure tree in sequence according to the second node relationship; wherein the same sample fields share nodes in the sample structure tree.

[0091] Specifically, for the sample field of a sample URL W1, the first sample field can be used as the root node of the sample structure tree, the second sample field can be used as the child node of the root node, and the subsequent sample fields can be added to the sample structure tree as the child nodes of the existing nodes in sequence to obtain the existing sample structure tree T. For the sample field of another sample URL W2, its first sample field is compared with the root node of the existing sample structure tree T; if they are inconsistent, the first sample field is used as a new node, and the subsequent sample fields are used as the child nodes of the new node; if they are consistent, the root node is shared, and the second sample field is compared with the nodes of the second level in the existing sample structure tree T. If they are inconsistent, it is used as a new node of the second level; if they are consistent, it is shared with the existing nodes of the second level. And so on, until the sample fields corresponding to each sample URL are added to the sample structure tree.

[0092] In one embodiment, the sample structure tree is constructed for each of the sample fields according to the second node relationship to obtain the sample structure tree, including: when the sample field contains a query field, each parameter value of the query field is used as a parallel leaf node in the sample structure tree.

[0093] In one embodiment, the query field is used to pass parameters to the dynamic web page, and these parameters are often in a parallel relationship, so all parameter names of the query field can be set as sub-stages of the last path, and the parameter values ​​are set as sub-nodes of the parameter names.

[0094] In one embodiment, assume the following sample URLs:

[0095] https: / / demo.qq.com / api / adddata / 1?key1=abc&key2=def

[0096] https: / / demo.qq.com / api / adddata / 2?key1=ghi&key2=jkm

[0097] Requirement: When merging nodes, the number of nodes in the sample structure tree must be greater than the node number threshold of 2.

[0098] The implementation process of building the target clustering structure tree based on the above sample URLs is described as follows:

[0099] 1. Split the sample URL into sample fields, arrange the sample fields together and separate them with commas. The result is as follows:

[0100] https:, demo.qq.com, api, adddata, 1, key1=abc, key2=def

[0101] https:, demo.qq.com, api, adddata, 2, key1=ghi, key2=jkm

[0102] 2. Construct a structure tree based on the sample fields. The resulting sample structure tree is as follows: Figure 3 shown.

[0103] 3. Pruning stage and export stage

[0104] Traversing the above sample structure tree layer by layer, there is only one "https:", "demo.qq.com", "api" and "adddata", which is less than the node number threshold of 2, and no pruning is required.

[0105] When traversing to the child nodes "1" and "2" of the "adddata" node, it is found that "adddata" has two child nodes, which is greater than or equal to the node number threshold 2, and can be pruned. Then the two nodes "1" and "2" are merged into one node "*", and the child nodes of these two nodes are adjusted (the same child nodes are integrated into one child node). The first pruning result is as follows Figure 4 shown.

[0106] Continue to scan the subsequent levels of the sample structure tree and find that the number of "key1" and "key2" nodes is 2, which is greater than or equal to the node number threshold 2. Pruning is possible, so the "key1" and "key2" nodes are merged into one node "*". The second pruning result is as follows: Figure 5 shown.

[0107] Continue to scan the sample structure tree and find that the number of nodes in the last level is 4, which is greater than or equal to the node number threshold 2. Pruning is possible, so these nodes are merged into one node "*". The third pruning result is as follows: Figure 6 shown.

[0108] 4. Figure 6 The structure tree in is determined as the target clustering structure tree. According to the target clustering structure tree, it can be determined that the above process clusters the two sample URLs into one cluster URL (URL category):

[0109] https: / / demo.qq.com / api / adddata / *?*=*

[0110] In the above embodiment, a sample structure tree is constructed based on sample URLs, and nodes in the sample structure tree are pruned layer by layer to obtain a target clustering structure tree after clustering.

[0111] In one embodiment, the obtaining of sample URLs includes: receiving the URLs for the target dynamic web pages sent by the front-end monitoring system; adding the URLs for the target dynamic web pages to a URL set, and performing duplicate URL removal processing on the URL set; when the number of URLs in the URL set is greater than or equal to a URL number threshold, determining that a clustering condition is met, and determining the URLs in the URL set as the sample URLs;

[0112] Among them, the front-end monitoring system (which can be referred to as the monitoring system) can be a website monitoring platform. The front-end monitoring system can obtain access requests from a specific application platform in real time. In some embodiments, the front-end monitoring system can be installed in the application platform; it can also be a system independent of the application platform, which obtains access requests in the application platform in a bypass manner, and then obtains the corresponding website. In one embodiment, the front-end monitoring system can be implemented by a terminal or a server.

[0113] A dynamic web page may refer to a web page that can interact with a user and responds to the user's interactive operation in a targeted manner, for example, when receiving a video playback trigger instruction from the user, the corresponding video is played in a new window. In one embodiment, a dynamic web page is a web page made using technologies such as CGI (Common Gateway Interface), ISAPI (Internet Server Application Programming Interface), PHP (Hypertext Preprocessor), JSP (Java Server Pages), ASP (Active Server Page), ASP.NET (also known as ASP+), etc.

[0114] In one embodiment, the server can obtain the URL in the following manner: Figure 7 As shown, the specific implementation process is as follows: the terminal sends an interaction request to the application platform to access a dynamic web page in the application platform; the front-end monitoring system extracts the URL from the interaction request sent to the application platform, and sends the extracted URL to the server; the server stores the received URL in a URL collection located in the cache database.

[0115] In one embodiment, the URL collection can store the acquired URLs according to the Set data structure. The Set data structure is a commonly used data structure in computer programming. This data structure uses the same mathematical concept as a finite set and is applied to computer data structures. It consists of a set of unordered and unique items, and generally provides functions such as adding items, deleting items, and counting item data. Furthermore, when the server stores the URL in the URL collection, the URL collection can automatically perform deduplication processing to ensure the uniqueness of the URLs in the URL collection.

[0116] In one embodiment, the server monitors the number of URLs in the URL set in real time, and determines that the clustering condition is met when the number of URLs is greater than or equal to the URL number threshold. At this time, all URLs in the URL set can be used as new sample URLs for clustering operations. If a clustered target clustering structure tree already exists in the cache database, the sample fields corresponding to these new sample URLs can be added to the target clustering structure tree to obtain a new clustering structure tree, and the nodes of the levels that meet the conditions in the new clustering structure tree are merged to update the clustering structure tree, so that the obtained clustering structure tree carries as much URL information as possible, thereby classifying the URLs to be classified more quickly and effectively.

[0117] The URL number threshold (maxURLCount) can be an empirical value, and a suitable value can be selected according to different application scenarios. In the front-end monitoring scenario, the number of URLs requested by a project usually does not exceed 1,000, so in this scenario, maxURLCount can be 1,000.

[0118] Traditionally, URLs are clustered using the K-means algorithm. The purpose of the K-means algorithm is to divide n points into k clusters so that each point belongs to the cluster corresponding to the nearest mean (i.e., the cluster center) as the clustering criterion. The main drawback of this algorithm is that the final number of clustering categories n needs to be specified in advance. Generally speaking, in a monitoring system, the number of URLs is generally transparent to the monitoring platform, and the monitoring platform cannot know the final number of clusters in advance. In order to achieve the best clustering effect, it is necessary to repeatedly try different n values ​​to explore the optimal number of clusters, resulting in performance waste.

[0119] The above embodiment stores the acquired URLs into a URL set, and automatically triggers clustering according to the number of URLs in the URL set. At the same time, there is no need to specify the number of clustering results in advance. The number of clustering results is automatically explored in the clustering. Compared with traditional technologies, the efficiency of URL clustering is greatly improved.

[0120] In one embodiment, when the first target level is accessed, the nodes in the first target level are merged into an arbitrary matching node, including: obtaining the identifier carrying information of each of the sample fields; the identifier carrying information is used to indicate whether the corresponding sample field carries a merge prohibition flag; when it is determined according to the identifier carrying information that the sample fields corresponding to each node in the current access level do not carry the merge prohibition flag, the number of nodes in the current access level is determined; the current access level is the node level in the target clustering structure tree currently accessed; when the number of nodes in the current access level is greater than or equal to a node number threshold, the current access level is determined to be the first target level, and the nodes in the current access level are merged into an arbitrary matching node.

[0121] When clustering the sample structure tree, the nodes of each level are visited layer by layer. For the currently visited level, the sample fields corresponding to each node in the level are obtained and the identification information corresponding to these sample fields is obtained. According to the identification information, it is determined whether the node carries a merge prohibition flag. If there is such a flag (that is, the node is marked as non-clusterable, or another way of saying it is marked as not being pruned), it means that the node cannot be merged.

[0122] In one embodiment, if there are more than one node in a certain level, and only some of the nodes are marked as non-clusterable, the other nodes in the level (nodes not marked as non-clusterable) can be clustered, and when the number of these nodes is greater than or equal to the node number threshold, these nodes are merged into an arbitrary matching node. Then, the subsequent levels are merged layer by layer, and so on. When the subsequent levels are merged, the child nodes of the nodes carrying the merge prohibition mark can participate or not participate.

[0123] Among them, important fields can be marked as non-clusterable. For important fields, on the one hand, important fields can be determined according to the number of occurrences of a certain field in each URL. When the number of occurrences of the field in each URL is greater than the set number threshold, it is considered to be an important field. For example, the https field appears in all URLs that apply the HTTP protocol and is an important field; on the contrary, when the number of occurrences is less than or equal to the set number threshold, it is considered to be a non-important field; of course, if the URL corresponding to the infrequently appearing field is to be analyzed, the field whose number of occurrences is less than or equal to the set number threshold can be determined as an important field. On the other hand, important fields can be fields that are more important from the perspective of URL analysis: 1. Fields such as protocol type, server address, port number, etc. that play an identification role for web pages can be important fields; 2. When it is necessary to analyze the response to the user request, the fields corresponding to the parameter name, parameter value, etc. related to the user interaction are important fields. In one embodiment, for a URL containing a query field, since the query field is often an important sign of whether the current operation request is a query operation, if the user's query request is to be analyzed, the query field is an important field.

[0124] In one embodiment, before obtaining the identification carrying information of each of the sample fields, it also includes: obtaining a set of historical URLs within a preset historical time period; determining the visit ratio of each URL in the set of historical URLs in the set of historical URLs; when there is a target URL in the set of historical URLs whose visit ratio is greater than a visit ratio threshold, determining a first target sample URL corresponding to the target URL in the sample URLs; and determining the identification carrying information corresponding to each sample field of the first target sample URL as a carrying merge prohibition identifier.

[0125] The inventors have found that when a URL has a high percentage of visits, clustering it using traditional URL similarity will lose this data. The embodiment of the present invention marks each node of the path corresponding to the target URL whose visit percentage is greater than a specific threshold as not to be pruned, which can achieve the retention of this data. That is, the embodiment of the present invention can separately classify URLs with sufficiently high visits to ensure that these URLs will not be submerged in other URLs during the URL analysis process. In this way, the response status of the URL can be analyzed more accurately, and then this type of URL can be processed in a timely manner when there is a problem, so as to better ensure the stable operation of the entire system.

[0126] There are two main ways to introduce parameters into a URL:

[0127] 1. Use the value of the parameter in [Query] to distinguish. For example:

[0128] www.getWeather.com / today / temperature?cityName=Beijing

[0129] www.getWeather.com / today / temperature?cityName=Shenzhen

[0130] Parameters are distinguished by parameter values ​​such as "Beijing" and "Shenzhen".

[0131] 2. Distinguish by [resource level UNIX file path]. For example:

[0132] www.getWeather.com / 20201013 / temperature?cityName=Beijing

[0133] www.getWeather.com / 20201014 / temperature?cityName=Beijing

[0134] Parameters are distinguished by [resource level UNIX file path].

[0135] Among them, [resource level UNIX file path] and [query] are constructed together into the URL structure tree. In fact, the format of [query] and UNIX file path is different.

[0136] For scenarios where URL analysis based on parameters is required. [Resource level UNIX file path] introduces parameters through paths. The order of parameters in the path determines the key (parameter name) of this parameter. It is not necessary to mark the parameter as non-clusterable. The key of the parameter in [query] determines the actual meaning of a parameter. The key of the [query] fragment is given directly, and the order does not affect it. Therefore, the key of the [query] fragment needs to be set to non-clusterable so that the parameter field is included in the obtained URL category.

[0137] In one embodiment, before obtaining the identifier carrying information, it also includes: determining a URL containing a query field in the sample URL as a second target sample URL; obtaining a target sample field corresponding to the query field of the second target sample URL in each of the sample fields; and determining the identifier carrying information of the target sample field as a carrying merge prohibition identifier.

[0138] The query field is often an important field that characterizes user interaction parameters. The embodiment of the present invention marks it as non-clusterable, which can effectively retain this important information and thus ensure the reliability of the URL classification results. That is, the URL is analyzed based on the user's interactive operation, and the response status of the application platform to the user's operation request can be determined in a targeted manner, which can effectively ensure the orderliness of interaction with the user.

[0139] In one embodiment, there are cases where nodes in the sample structure tree are marked as non-clusterable, for example, Figure 3 In the sample structure tree shown, the sample fields corresponding to the "key1" and "key2" nodes are marked as non-clusterable. The clustering process based on the sample structure tree can be as follows:

[0140] Traversing the above sample structure tree layer by layer, there is only one "https:", "demo.qq.com", "api" and "adddata", which is less than the node number threshold of 2, and no pruning is required.

[0141] When traversing to the child nodes of the "adddata" node, it is found that "adddata" has two child nodes, which is greater than or equal to the node number threshold 2, and can be pruned. Then the two nodes "1" and "2" are merged into one node "*", and the child nodes of these two nodes are adjusted. The first pruning result is as follows Figure 4 shown.

[0142] Continuing to scan the subsequent levels of the sample structure tree, the number of nodes of "key1" and "key2" is 2, which can be clustered. However, the sample fields corresponding to these two nodes are marked as non-clusterable, so clustering is not performed at this level.

[0143] Continue to scan the subsequent levels of the sample structure tree and find that the number of child nodes of the "key1" and "key2" nodes are both 2. Pruning is possible. The child nodes of the "key1" and "key2" nodes are merged into one node "*". The second pruning result is as follows: Figure 8 shown.

[0144] Continue to scan the sample structure tree and find that there is no child node. Figure 8 The structure tree in is determined as the target clustering structure tree, and the URL categories corresponding to the target clustering structure tree are as follows:

[0145] https: / / demo.qq.com / api / adddata / *?key1=*&key2=*

[0146] The above embodiment constructs a sample structure tree and prunes the sample structure tree layer by layer, incorporating some important performance data into the clustering index range, retaining important nodes marked as non-clusterable during clustering, implementing a weighted convergence pruning strategy, and obtaining a simplified target clustering structure tree. The clustering result has more practical reference significance.

[0147] In one embodiment, the embodiment of the present invention controls the degree of clustering through a node number threshold (maxChlidCount). If the number of nodes in the current access level is greater than or equal to the node number threshold, it is considered that the number of nodes in the level is too large, and these nodes are merged into an arbitrary matching node.

[0148] The node quantity threshold can be determined by the following embodiments:

[0149] In one embodiment, when the number of nodes in the current access level is greater than or equal to a node number threshold, before determining that the current access level is the first target level, it also includes: obtaining a preset proportional coefficient; obtaining the number of URLs of the sample URLs, and calculating the product of the number of URLs and the proportional coefficient; and determining the product as the node number threshold.

[0150] Among them, the proportional coefficient can be set according to the requirements of the degree of URL clustering. For example, when the URLs need to be clustered into as few URL categories as possible, the proportional coefficient can be set to a larger value, such as 20%. When the URLs need to be clustered into the most accurate URL categories, the proportional coefficient can be set to a smaller value, such as 10%.

[0151] In the previous embodiment, the node quantity threshold value varies with the number of sample URLs. In some embodiments, the node quantity threshold value may also be a fixed value, for example, fixed to 20.

[0152] In the above embodiment, a node quantity threshold is set. When the nodes at a certain level are not marked as non-clusterable and the number of nodes is greater than or equal to the node quantity threshold, the nodes at this level are aggregated into one node. In the subsequent process of classifying the URLs, the clustered node can match all nodes. In this way, a large number of URLs are clustered into one category, which greatly simplifies the computational complexity of the URL analysis process.

[0153] In one embodiment, the acquisition of the target clustering structure tree includes: determining the expiration time of a pre-constructed clustering structure tree to be selected; when the expiration time has not been reached, determining the clustering structure tree to be selected as the target clustering structure tree; when the expiration time is reached, acquiring a new sample structure tree, and merging the nodes of the third target level whose number of nodes in the new sample structure tree meets the condition to obtain the target clustering structure tree.

[0154] In one embodiment, when the server obtains the target clustering structure tree by clustering, the target clustering structure tree can be stored in the cache database, making it convenient for multiple machines to share. Now, the expiration time can be set for the target clustering structure tree, and the cache database saves the target clustering structure tree when the expiration time does not arrive, and releases the target clustering structure tree when the expiration time arrives. If you receive the website to be classified and do not reach the expiration time, then the server can carry out the website classification based on the target clustering structure tree in the cache database, if you receive the website to be classified and reach the expiration time, can regenerate the clustering structure tree. Wherein, regenerating the clustering structure tree is the same as the mode of generating the target clustering structure tree, and will not repeat them here.

[0155] The above embodiment sets an expiration time for the target clustering structure tree, which can effectively reduce the storage pressure of the cache database.

[0156] In one embodiment, the process of generating a clustering structure tree, website classification and website analysis can be performed in parallel. For example, the server receives the website addresses sent by the front-end monitoring system and stores these website addresses in the website address collection. When clustering, the server first obtains these website addresses, clusters these website addresses to obtain the target clustering structure tree. Afterwards, website classification and website analysis can be performed based on the target clustering structure tree: on the one hand, if there are new website addresses in the website address collection, these website addresses are read one by one and used as website addresses to be classified, and the website addresses to be classified are matched with the target clustering structure tree to obtain the website address categories corresponding to these website addresses to be classified. On the other hand, when the server receives the website address response status analysis instruction sent by the terminal, it can analyze the matching website addresses of the corresponding website address categories according to the target clustering structure tree to output the corresponding website address response status analysis results.

[0157] The above embodiment can quickly complete the clustering of URLs within a few minutes. After the clustering is completed, the remaining URLs to be classified can be immediately classified according to the clustering structure tree, and the clustering structure tree can be used for data statistics and analysis. It can greatly improve the efficiency of URL classification and URL analysis.

[0158] In one embodiment, after determining the URL category of the URL to be classified according to the target path, the method further includes: when receiving a URL response status analysis instruction for a target public gateway interface, determining a path identifier corresponding to the target public gateway interface; determining a target URL category corresponding to the path identifier; determining a target matching URL that matches the target URL category; obtaining response status data corresponding to the target matching URL; and generating a URL response status analysis result of the target public gateway interface according to the response status data. Wherein, the public gateway interface is a CGI interface.

[0159] In one embodiment, the URL response status may refer to the response status of the request corresponding to the URL, and may refer to the request success rate, request speed, response consumption time, etc.

[0160] In one embodiment, the terminal displays the following in the interface: Fig. 9 In the window shown, the user can enter the URL search information in the window. Specifically, the user can enter the URL search keyword in the "Search URL" option, such as: Beijing's weather, or directly enter the URL category, such as: www.getWeather.com / today / temperature? cityName=Beijing. In addition, the user can select the time range and time granularity in the window to conduct a specific fine-grained analysis of the URL in a specific time period. When the terminal receives the URL search information entered by the user through the window, the URL search information is carried in the URL response status analysis instruction and sent to the server. The server determines the path identifier corresponding to the Beijing weather interface; determines the target URL category corresponding to the path identifier; determines the target matching URL that matches the target URL category; obtains the response status data corresponding to the target matching URL; generates the URL response status analysis result of the target public gateway interface according to the response status data, and returns the URL response status analysis result to the terminal.

[0161] The server can control the terminal to display the website response status analysis results on the terminal page in various intuitive graphical forms, such as wave charts, bar charts, pie charts, etc.

[0162] Specifically, a schematic diagram of gradually clustering and classifying URLs and outputting URL response status analysis results can be shown as follows: Fig.10 shown. Fig.10 The number of samples 1001 increases gradually, that is, URLs are read from the URL set step by step and analyzed; in addition, the average response speed 1002 to these URLs remains in a relatively stable state.

[0163] In addition, you can also Fig.11The graph shown in the figure shows the results of the URL response status analysis. Specifically, the distribution of request speed is displayed in the form of a bar graph, and the distribution of response return codes is displayed in the form of a pie chart. At the same time, the request success rate can be displayed in the interface.

[0164] It should be noted that, as the read URL changes, the URL response status analysis result in the interface may change in real time, so that the user can obtain the corresponding URL response status analysis result in a timely manner.

[0165] When receiving a URL response status analysis instruction for a certain CGI interface, this embodiment specifically obtains the target matching URL based on the previous URL classification result without searching for URLs one by one, which can greatly improve the analysis efficiency of the URL response status.

[0166] The present application also provides an application scenario, which applies the above-mentioned URL classification method. Specifically, the method is applied to Figure 1 Take the server in as an example to illustrate, the application of the URL classification method in this application scenario is as follows:

[0167] 1. For a request to obtain weather information of a specific city sent to the application platform, due to the introduction of the variable parameter cityName, the website uploaded 1 million URLs with different parameters in one day. When it is necessary to analyze these 1 million URLs, if the 1 million URLs are analyzed directly, the amount of calculation is very large, so it is necessary to classify the URLs and analyze them based on the classified URLs.

[0168] 2. Assume that the URL collection contains the following two sample URLs in the form of a Set data structure:

[0169] www.getWeather.com / today / temperature?cityName=Beijing

[0170] www.getWeather.com / today / wind?cityName=Shenzhen

[0171] maxURLCount is set to 2 (for ease of description, it is set to a smaller value, and can be a larger value in actual application scenarios). In addition, the preset proportionality factor is 1, and the product of the proportionality factor and the number of sample URLs is 2 as maxChlidCount.

[0172] When the number of URLs in the URL set is greater than or equal to maxURLCount, it is considered necessary to perform URL clustering analysis and enter the next stage.

[0173] 3. URL structure tree construction stage

[0174] Take all URLs in the URL set and split each URL using the " / ", "&", and "?" in the URL as delimiters to obtain the following sample fields:

[0175] www.getWeather.com, today, temperature, cityName = Beijing

[0176] www.getWeather.com, today, wind, cityName = Shenzhen

[0177] After the split, you can start traversing the split fields and build a URL structure tree. The first field is the root node, and the next field is the child node of the previous field. If the subsequent fields are the same, no new nodes are generated, and the subsequent fields are directly set as child nodes of the existing fields. For the [query] string (cityName field), all parameter names are set as the child stages of the last path, the parameter values ​​are set as the child nodes of the parameter name, and the parameter name node "cityName" is marked as not to be pruned.

[0178] 4. Pruning stage

[0179] After the tree is built successfully, a depth-first scan is performed on the structure tree. When the number of child nodes of a node is greater than or equal to maxChlidCount and these child nodes can be pruned, all child nodes of the node are merged into a single node and marked with *. The next-level child nodes of these child nodes are also merged into the * node.

[0180] 5. Export stage

[0181] Repeat step 4 until there is no level in the structure tree where the number of nodes exceeds maxChildCount. At this point, the clustered URLs can be obtained, and the following two URL categories can be obtained:

[0182] www.getWeather.com / today / *?cityName=*

[0183] www.getWeather.com / today / *?cityName=*

[0184] The clustering structure tree is then stored in a cache database to facilitate sharing by multiple machines, and an expiration time is set, after which it will be recalculated.

[0185] 6. Classification stage

[0186] After clustering is completed, when there is a new URL that needs to be classified, the clustering structure tree is synchronized from the cache database at a regular time. After the new URL is segmented by the flag bit, these segmented fields are directly matched with the clustering structure tree, and the * node matches any field. When the last field matches the leaf node of the structure tree, it is considered that the match is successful and the classification result is obtained. Otherwise, the classification fails, and the new URL continues to be inserted into the constructed tree and waits for the next clustering.

[0187] In the above embodiment, based on the tree data structure, a structure tree is constructed and pruned for the user request URLs collected in the front-end monitoring system, thereby achieving URL clustering. After clustering, subsequent URL analysis can directly use the structure tree for matching and classification.

[0188] The present application also provides an application scenario, which applies the above-mentioned URL classification method. Specifically, the method is applied to Figure 1 Take the server in as an example to illustrate, the application of the URL classification method in this application scenario is as follows:

[0189] Taking Web front-end monitoring as an example, in order to continuously optimize the speed and experience of Web applications, the monitoring platform is used to collect and analyze the user's HTTP request data. Taking CGI requests as an example, after collecting all the original CGI request data of users, the request data is analyzed according to the dimension of URL, so as to better identify the shortcomings in the application and make targeted optimizations. The specific implementation process can be:

[0190] 1. The terminal sends various types of CGI requests to the weather website. The front-end monitoring system monitors these CGI requests in real time, selects the CGI requests for obtaining weather information, extracts the URLs, and sends the extracted URLs to the server. The server stores the received URLs in the URL collection in the form of a Set data structure. According to the analysis of CGI requests, it can be found that the number of requests for Beijing is much greater than that for other cities. We want to analyze the performance data under the scenario of cityName = "Beijing" separately.

[0191] 2. Assume that the following four sample URLs are stored in the URL collection:

[0192] www.getWeather.com / today / temperature?cityName=Beijing

[0193] www.getWeather.com / today / temperature?cityName=Shenzhen

[0194] www.getWeather.com / today / wind?cityName=Shanghai

[0195] www.getWeather.com / today / wind?cityName=Guangzhou

[0196] Among them, maxURLCount is set to 2 and maxChlidCount is set to 2.

[0197] The number of URLs in the URL set is 2, which is equal to maxURLCount. It is considered necessary to perform URL clustering analysis and proceed to the next stage.

[0198] 3. URL structure tree construction stage

[0199] Take all URLs in the URL collection and split each URL by the delimiter to get the following sample fields:

[0200] www.getWeather.com, today, temperature, cityName = Beijing

[0201] www.getWeather.com, today, temperature, cityName = Shenzhen

[0202] www.getWeather.com, today, wind, cityName = Shanghai

[0203] www.getWeather.com, today, wind, cityName = Guangzhou

[0204] After the split, you can start traversing the cut fields and build a clustering structure tree. Among them, "cityName" and "Beijing" are marked as not to be pruned.

[0205] 4. Pruning stage

[0206] After the structure tree is constructed successfully, a depth-first scan is performed on the structure tree. When the number of child nodes of a node is greater than or equal to 2 and the node is not marked as unpruneable, all child nodes of the node are merged into a single node and marked with *. The next-level child nodes of these child nodes are also merged into the * node.

[0207] 5. Export stage

[0208] Repeat step 4 until there is no level in the structure tree where the number of nodes exceeds maxChildCount. At this point, the cluster structure tree can be obtained by deep traversal. The cluster structure tree is then stored in the cache database.

[0209] The URL categories corresponding to the clustering structure tree can be as follows:

[0210] www.getWeather.com / today / *?cityName=Beijing

[0211] www.getWeather.com / today / *?cityName=*

[0212] 6. Classification stage

[0213] After clustering is completed, when there is a new URL that needs to be classified, the clustering structure tree is synchronized from the cache database at a regular interval.

[0214] Assume the new URL is:

[0215] www.getWeather.com / today / humidity?cityName=Beijing

[0216] After field segmentation, the following target fields are obtained:

[0217] www.getWeather.com, today, humidity, cityName = Beijing

[0218] The matching of each target field with the nodes in the clustering structure tree is judged layer by layer. According to the judgment result, it is determined that each target field can match each node in the path of "www.getWeather.com→today→*→cityName=Beijing".

[0219] Therefore, the new URL can be classified into the URL category of "www.getWeather.com / today / *?cityName=Beijing".

[0220] 7. URL analysis phase

[0221] If a URL response status analysis instruction for Beijing weather is received from the terminal, the target matching URL that matches "www.getWeather.com / today / *?cityName=Beijing" is obtained. The request success rate, request speed, and response consumption time corresponding to these target matching URLs are determined. The determined request success rate, request speed, and response consumption time are returned to the terminal as the URL response status analysis result, so that the terminal displays the URL response status analysis result on the interface.

[0222] The above embodiment, which classifies new URLs based on a pre-constructed clustering structure tree, has at least the following effects:

[0223] 1) It will automatically determine the timing of clustering, without the need for pre-configuration of parameters.

[0224] 2) Due to the adoption of a progressive clustering strategy, when faced with massive URL time series data, this solution can complete the construction of the URL classification tree in a shorter cold start time, start clustering the subsequently arriving URLs, and continuously improve the URL structure tree, thereby improving the classification accuracy.

[0225] 3) It has very good performance when facing massive URL classification tasks. For example, in a test on Tencent Cloud's 8-core 16G cloud virtual machine, after building the URL structure tree, this solution can complete the classification task of more than 500,000 URLs within 1 second.

[0226] 4) Since users access Web applications continuously, this solution is particularly suitable for user monitoring in front-end projects.

[0227] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0228] Based on the same idea as the URL classification method in the above embodiment, the present invention also provides a URL classification device, which can be used to execute the above URL classification method. For ease of explanation, the structural diagram of the URL classification device embodiment only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0229] In one embodiment, Fig.12 As shown, a device 1200 for classifying a website is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a website to be classified acquisition module 1201, a structure tree acquisition module 1202, a layer-by-layer matching module 1203 and a website category determination module 1204, wherein:

[0230] The module 1201 for obtaining the website address to be classified is used to obtain the website address to be classified, and to obtain the target field corresponding to the first node relationship after performing field segmentation on the website address to be classified.

[0231] The structure tree acquisition module 1202 is used to acquire a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose node quantity in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs.

[0232] The layer-by-layer matching module 1203 is used to match each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship.

[0233] The website category determination module 1204 is used to determine the website category of the website to be classified according to the target path when each of the target fields matches each node in the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

[0234] In the above-mentioned URL classification device, the URLs to be classified can be directly classified based on the pre-constructed target clustering structure tree, and the URL classification results of the URLs to be classified can be quickly output, thereby effectively ensuring the efficiency of URL classification.

[0235] In one embodiment, the layer-by-layer matching module includes: a layer-by-layer matching submodule, which is used to determine the nodes that match each of the target fields layer by layer in the target clustering structure tree according to the first node relationship to obtain the target node; and a matching judgment submodule, which is used to judge that each of the target fields matches each of the nodes in the target path when each of the target nodes constitutes a path from a root node to a leaf node in the target clustering structure tree.

[0236] In one embodiment, the layer-by-layer matching module also includes: a field superposition submodule, which is used to superimpose each of the target fields into the target clustering structure tree according to the first node relationship when each of the target nodes cannot form a path from the root node to the leaf node in the target clustering structure tree, so as to obtain a structure tree to be merged; a structure tree update submodule, which is used to merge the nodes of the second target level whose number of nodes in the structure tree to be merged meets the conditions when a clustering instruction is obtained, so as to obtain a clustering update structure tree; the clustering update structure tree is used to classify the new URL to be classified when the new URL to be classified is obtained.

[0237] In one embodiment, the URL classification device also includes: a sample URL acquisition module, used to acquire sample URLs, and obtain sample fields corresponding to the second node relationship after field segmentation of the sample URLs; a structure tree construction module, used to construct a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree; a node merging module, used to perform node merging operations on the sample structure tree layer by layer, so as to merge the nodes in the first target level into an arbitrary matching node when the first target level is accessed, and merge the child nodes of each node in the first target level into the child nodes of the arbitrary matching node; the arbitrary matching node can match any node; a structure tree determination module, used to use the sample structure tree after node merging as the target clustering structure tree.

[0238] In one embodiment, the sample URL acquisition module includes: a URL receiving submodule, which is used to receive the URL for the target dynamic web page sent by the front-end monitoring system; a URL adding submodule, which is used to add the URL for the target dynamic web page to a URL set and remove duplicate URLs from the URL set; and a sample URL determination submodule, which is used to determine that the clustering condition is met when the number of URLs in the URL set is greater than or equal to a URL number threshold, and determine the URLs in the URL set as the sample URLs.

[0239] In one embodiment, the structure tree construction module is further used to sequentially configure each of the sample fields as a node in the sample structure tree according to the second node relationship; wherein the same sample fields share nodes in the sample structure tree.

[0240] In one embodiment, a node merging module includes: an information acquisition submodule, used to acquire the identifier carrying information of each of the sample fields; the identifier carrying information is used to indicate whether the corresponding sample field carries a merge prohibition flag; a node quantity acquisition submodule, used to determine the number of nodes in the current access level when it is determined according to the identifier carrying information that the sample fields corresponding to each node in the current access level do not carry the merge prohibition flag; the current access level is the node level in the target clustering structure tree currently accessed; a node merging submodule, used to determine that the current access level is the first target level when the number of nodes in the current access level is greater than or equal to a node quantity threshold, and merge each node in the current access level into an arbitrary matching node.

[0241] In one embodiment, the node merging module also includes: a URL set acquisition submodule, used to acquire a historical URL set within a preset historical time period; a visit volume ratio acquisition submodule, used to determine the visit volume ratio of each URL in the historical URL set in the historical URL set; a first target URL determination submodule, used to determine a first target sample URL corresponding to the target URL in the sample URLs when there is a target URL in the historical URL set whose visit volume ratio is greater than a visit volume ratio threshold; a first information determination submodule, used to determine the identifier carrying information corresponding to each sample field of the first target sample URL as a carrying merging prohibition identifier.

[0242] In one embodiment, the node merging module also includes: a second target URL determination submodule, used to determine the URL containing the query field in the sample URL as the second target sample URL; a field determination submodule, used to obtain the target sample field corresponding to the query field of the second target sample URL in each of the sample fields; and a second information determination submodule, used to determine the identification carrying information of the target sample field as a carrying merging prohibition identification.

[0243] In one embodiment, the node merging module further includes: a proportional coefficient acquisition submodule, used to obtain a preset proportional coefficient; a product calculation submodule, used to obtain the number of URLs of the sample URLs and calculate the product of the number of URLs and the proportional coefficient; and a quantity threshold determination submodule, used to determine the product as the node quantity threshold.

[0244] In one embodiment, a structure tree acquisition module includes: an expiration time determination submodule, used to determine the expiration time of a pre-constructed clustering structure tree to be selected; a first structure tree determination submodule, used to determine the clustering structure tree to be selected as the target clustering structure tree when the expiration time has not been reached; and a second structure tree determination submodule, used to acquire a new sample structure tree when the expiration time is reached, and merge the nodes of the third target level whose node quantity in the new sample structure tree meets the condition to obtain the target clustering structure tree.

[0245] In one embodiment, the URL classification device also includes: a path identifier determination module, which is used to determine the path identifier corresponding to the target public gateway interface when receiving a URL response status analysis instruction for the target public gateway interface; a target URL category determination module, which is used to determine the target URL category corresponding to the path identifier; a matching URL determination module, which is used to determine a target matching URL that matches the target URL category; a status data acquisition module, which is used to acquire response status data corresponding to the target matching URL; and an analysis result determination module, which is used to generate a URL response status analysis result of the target public gateway interface according to the response status data.

[0246] The specific definition of the URL classification device can be found in the definition of the URL classification method above, which will not be repeated here. Each module in the above URL classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0247] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a structure tree and a collection of website addresses. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for classifying website addresses is implemented.

[0248] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0249] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0250] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0251] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0252] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0253] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for classifying URLs, characterized in that: The method comprises: Obtaining a website to be classified, and performing field segmentation on the website to be classified to obtain a target field corresponding to the first node relationship; Determine the expiration time of the pre-built candidate clustering structure tree; When the expiration time has not arrived, the candidate clustering structure tree is determined as the target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the condition, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs; When the expiration time arrives, a new sample structure tree is obtained, and nodes of the third target level whose number of nodes in the new sample structure tree meets the condition are merged to obtain the target clustering structure tree; Matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship; When each of the target fields matches each of the nodes in the target path, the URL category of the URL to be classified is determined according to the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

2. The method according to claim 1, characterized in that The step of matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship includes: Determine the nodes matching the target fields layer by layer in the target clustering structure tree according to the first node relationship to obtain the target node; When each of the target nodes forms a path from a root node to a leaf node in the target clustering structure tree, it is determined that each of the target fields matches each of the nodes in the target path.

3. The method according to claim 2, characterized in that After determining the nodes matching the target fields layer by layer in the target clustering structure tree according to the first node relationship and obtaining the target nodes, the method further includes: When each of the target nodes cannot form a path from the root node to the leaf node in the target clustering structure tree, each of the target fields is superimposed into the target clustering structure tree according to the first node relationship to obtain a structure tree to be merged; When a clustering instruction is obtained, the nodes of the second target level whose number of nodes in the structure tree to be merged meets the condition are merged to obtain a clustering update structure tree; the clustering update structure tree is used to classify the new URL to be classified when the new URL to be classified is obtained.

4. The method according to claim 1, characterized in that: The method further comprises: Obtain a sample URL, and perform field segmentation on the sample URL to obtain a sample field corresponding to the second node relationship; Constructing a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree; Performing a node merging operation on the sample structure tree layer by layer, so that when the first target level is accessed, the nodes in the first target level are merged into an arbitrary matching node, and the child nodes of each node in the first target level are merged into the child nodes of the arbitrary matching node; the arbitrary matching node can match any node; The sample structure tree after merging the nodes is used as the target clustering structure tree.

5. The method according to claim 4, characterized in that The obtaining of the sample URL includes: Receive the URL of the target dynamic web page sent by the front-end monitoring system; Adding the URL for the target dynamic web page to a URL set, and performing a process of removing duplicate URLs from the URL set; When the number of URLs in the URL set is greater than or equal to the URL number threshold, it is determined that the clustering condition is met, and the URLs in the URL set are determined as the sample URLs.

6. The method according to claim 4, characterized in that The step of constructing a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree includes: The sample fields are sequentially configured as nodes in the sample structure tree according to the second node relationship; wherein the same sample fields share nodes in the sample structure tree.

7. The method according to claim 4, characterized in that The step of merging the nodes in the first target level into an arbitrary matching node when the first target level is accessed includes: Acquire the identifier-carrying information of each of the sample fields; the identifier-carrying information is used to indicate whether the corresponding sample field carries a merging prohibition identifier; When it is determined according to the information carried by the identifier that the sample field corresponding to each node in the current access level does not carry the merge prohibition identifier, the number of nodes in the current access level is determined; the current access level is the node level in the target clustering structure tree currently accessed; When the number of nodes in the current access level is greater than or equal to a node number threshold, the current access level is determined to be the first target level, and each node in the current access level is merged into an arbitrary matching node.

8. The method according to claim 7, characterized in that Before obtaining the identification information of each sample field, the method further includes: Get the historical URL collection within the preset historical time period; Determine the visit ratio of each URL in the historical URL set in the historical URL set; When there is a target URL whose visit volume ratio is greater than a visit volume ratio threshold in the historical URL set, determining a first target sample URL corresponding to the target URL in the sample URLs; The identification carrying information corresponding to each sample field of the first target sample website is determined as a carrying merging prohibition identification.

9. The method according to claim 7, characterized in that: The method further comprises: Determine a URL containing a query field in the sample URL as a second target sample URL; Obtaining the target sample field corresponding to the query field of the second target sample website in each of the sample fields; The identifier carrying information of the target sample field is determined to be a carrying merging prohibition identifier.

10. The method according to claim 7, characterized in that When the number of nodes at the current access level is greater than or equal to a node number threshold, before determining that the current access level is the first target level, the method further includes: Get the preset scale factor; Obtaining the number of URLs of the sample URLs, and calculating the product of the number of URLs and the proportional coefficient; The product is determined as the node quantity threshold.

11. The method according to any one of claims 1 to 10, characterized in that: After determining the website category of the website to be classified according to the target path, the method further includes: When receiving a URL response status analysis instruction for a target public gateway interface, determining a path identifier corresponding to the target public gateway interface; Determine a target website category corresponding to the path identifier; Determine a target matched URL that matches the target URL category; Obtaining response status data corresponding to the target matching URL; Generate a website address response status analysis result of the target CGI according to the response status data.

12. A device for classifying web addresses, characterized in that: The device comprises: A module for obtaining a website address to be classified, used for obtaining a website address to be classified, and performing field segmentation on the website address to be classified to obtain a target field corresponding to the first node relationship; A structure tree acquisition module is used to determine the expiration time of a pre-constructed clustering structure tree to be selected; when the expiration time has not been reached, the clustering structure tree to be selected is determined as a target clustering structure tree; the target clustering structure tree is obtained by merging the nodes of the first target level whose number of nodes in the sample structure tree meets the conditions, and the sample structure tree is a structure tree constructed according to the fields corresponding to the sample URLs; when the expiration time is reached, a new sample structure tree is obtained, and the nodes of the third target level whose number of nodes in the new sample structure tree meets the conditions are merged to obtain the target clustering structure tree; A layer-by-layer matching module, used for matching each of the target fields with each of the nodes in the target clustering structure tree layer by layer according to the first node relationship; The website category determination module is used to determine the website category of the website to be classified according to the target path when each of the target fields matches each node in the target path; the target path is the path from the root node to the leaf node in the target clustering structure tree.

13. The device according to claim 12, characterized in that The layer-by-layer matching module comprises: A layer-by-layer matching submodule, used to determine the nodes matching each of the target fields layer by layer in the target clustering structure tree according to the first node relationship, to obtain a target node; The matching determination submodule is used to determine whether each of the target fields matches each of the nodes in the target path when each of the target nodes forms a path from a root node to a leaf node in the target clustering structure tree.

14. The device according to claim 13, characterized in that The layer-by-layer matching module further includes: A field superposition submodule is used for superimposing each of the target fields into the target clustering structure tree according to the first node relationship to obtain a structure tree to be merged when each of the target nodes cannot form a path from a root node to a leaf node in the target clustering structure tree; The structure tree update submodule is used to merge the nodes of the second target level whose number of nodes in the structure tree to be merged meets the conditions when a clustering instruction is obtained, so as to obtain a cluster update structure tree; the cluster update structure tree is used to classify the new website to be classified when a new website to be classified is obtained.

15. The device according to claim 12, characterized in that The device also includes: A sample website acquisition module, used to acquire a sample website, and to obtain a sample field corresponding to a second node relationship after segmenting the sample website into fields; A structure tree construction module, used to construct a structure tree for each of the sample fields according to the second node relationship to obtain the sample structure tree; A node merging module, configured to perform node merging operations on the sample structure tree layer by layer, so as to merge the nodes in the first target level into an arbitrary matching node when the first target level is accessed, and merge the child nodes of each node in the first target level into the child nodes of the arbitrary matching node; the arbitrary matching node can match any node; The structure tree determination module is used to use the sample structure tree after the nodes are merged as the target clustering structure tree.

16. The device according to claim 15, characterized in that The sample website acquisition module includes: The website receiving submodule is used to receive the website address of the target dynamic web page sent by the front-end monitoring system; A website adding submodule, used for adding the website address for the target dynamic web page to the website address set, and performing a process of removing duplicate website addresses from the website address set; The sample website address determination submodule is used to determine that the clustering condition is met when the number of websites in the website address set is greater than or equal to the website address number threshold, and determine the websites in the website address set as the sample websites.

17. The device according to claim 15, characterized in that The structure tree construction module is further used to sequentially configure each of the sample fields as a node in the sample structure tree according to the second node relationship; wherein the same sample fields share nodes in the sample structure tree.

18. The device according to claim 15, characterized in that The node merging module includes An information acquisition submodule, used to acquire the identifier-carrying information of each of the sample fields; the identifier-carrying information is used to indicate whether the corresponding sample field carries a merging prohibition identifier; A node quantity acquisition submodule, used to determine the number of nodes in the current access level when it is determined according to the information carried by the identifier that the sample field corresponding to each node in the current access level does not carry the merge prohibition identifier; The currently accessed level is the node level in the target clustering structure tree currently accessed; The node merging submodule is used to determine that the current access level is the first target level when the number of nodes in the current access level is greater than or equal to a node number threshold, and merge the nodes in the current access level into an arbitrary matching node.

19. The device according to claim 18, characterized in that The node merging module also includes: A website collection acquisition submodule is used to acquire a historical website collection within a preset historical time period; A visit volume proportion acquisition submodule is used to determine the visit volume proportion of each URL in the historical URL set in the historical URL set; A first target website address determination submodule, configured to determine a first target sample website address corresponding to the target website address from among the sample websites when there is a target website address in the historical website address set whose visit volume ratio is greater than a visit volume ratio threshold; The first information determination submodule is used to determine the identification carrying information corresponding to each sample field of the first target sample website as a carrying merging prohibition identification.

20. The device according to claim 18, characterized in that The node merging module further includes: A second target website address determination submodule, used to determine a website address containing a query field in the sample website addresses as a second target sample website address; A field determination submodule, used for obtaining the target sample field corresponding to the query field of the second target sample website in each of the sample fields; The second information determination submodule is used to determine the identifier carrying information of the target sample field as a carrying merging prohibition identifier.

21. The device according to claim 18, characterized in that The node merging module also includes: A scale factor acquisition submodule is used to obtain a preset scale factor; A product calculation submodule, used for obtaining the number of URLs of the sample URLs, and calculating the product of the number of URLs and the proportional coefficient; The quantity threshold determination submodule is used to determine the product as the node quantity threshold.

22. The device according to any one of claims 12 to 21, characterized in that The device also includes: A path identifier determination module, configured to determine a path identifier corresponding to a target public gateway interface when receiving a URL response status analysis instruction for the target public gateway interface; A target website category determination module, used to determine the target website category corresponding to the path identifier; A matching URL determination module, used to determine a target matching URL that matches the target URL category; A status data acquisition module, used to acquire the response status data corresponding to the target matching URL; The analysis result determination module is used to generate a website response status analysis result of the target public gateway interface according to the response status data.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images