Method and device for generating page identifier

Abstract

The invention relates to a method and a device for generating a page identifier. The method comprises the following steps: acquiring page data, wherein the page data comprises a page address and page content; the method further includes determining a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to the user and a second query parameter unrelated to page content. In addition, the method further includes generating a page identifier based on the target portion. In this way, normalization processing can be carried out on a plurality of pages with different page addresses and the same page content, so that the pages with the same page content have unique page identifiers. Therefore, under application scenes of webpage indexing, data statistical analysis, webpage data caching and security monitoring, webpages with the same page content of multiple different network addresses and related data of the webpages can be identified, repeated content can be eliminated, processing efficiency can be improved, forged page address attacks can be prevented, and system security can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method and device for generating a page identifier. Background Art

[0002] As the number of web pages on the Internet increases, more and more duplicate web pages appear on the Internet. For example, the same web page has multiple different URLs, resulting in a large number of duplicate web pages with the same web page content but different URLs (Uniform Resource Locators). The existence of a large number of duplicate web pages not only increases the burden of search engines to crawl, index and store web pages, but is also not conducive to data analysis and statistical processing, and the processing efficiency is greatly reduced. In addition, forged URLs hidden between multiple duplicate web pages increase the difficulty of system security identification. Summary of the invention

[0003] The embodiments of the present disclosure provide a method and device for generating a page identifier.

[0004] In a first aspect of the present disclosure, a method for generating a page identifier is provided. The method includes acquiring page data, the page data including a page address and page content. The method also includes determining a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to a user and a second query parameter unrelated to the page content. In addition, the method also includes generating a page identifier based on the target portion.

[0005] In a second aspect of the present disclosure, a device for generating a page identifier is provided. The device includes an acquisition module configured to acquire page data, the page data including a page address and page content. The device also includes a first determination module configured to determine a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to the user and a second query parameter unrelated to the page content. In addition, the device also includes a generation module configured to generate a page identifier based on the target portion.

[0006] In a third aspect of the present disclosure, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the method according to the first aspect.

[0007] In a fourth aspect of the present disclosure, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, wherein the machine-executable instructions are executed by a processor to implement the method provided according to the first aspect of the present disclosure.

[0008] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which some embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flow chart showing a method for generating a page identifier according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of a process for determining a first query parameter and a second query parameter according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing an example process of determining a first query parameter and a second query parameter according to some embodiments of the present disclosure;

[0014] Figure 5 A flowchart of a page content understanding process according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A block diagram showing an apparatus for generating a page identifier according to some embodiments of the present disclosure; and

[0016] Figure 7 A block diagram of an electronic device in which various embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] As mentioned above, there are more and more duplicate web pages on the Internet. For example, web pages under the same domain name contain a large number of irrelevant variable parameters, resulting in multiple different URLs (hereinafter referred to as page addresses) for the same web page. A large number of duplicate web pages increases the system burden in various application scenarios such as data processing, indexing, storage, and security protection, and reduces system processing efficiency.

[0020] The URL usually consists of the protocol, domain name and port, path, query parameter, and hash value. For example, the URL is http: / / www.happymmall.com:80 / get_data?productid=1#title, where "http" is the protocol, and https, ftp, etc. are also protocols that specify the access method of the resource. "www.happymmall.com" is the domain name, indicating the server where the resource is located, ":80" is the port connected to the server, " / get_data.do" is the path, indicating the chicken leg location of the resource on the server, "?productid=1" is the query parameter. When there are multiple query parameters, they are generally connected with &, and "#title" is the hash value.

[0021] In some related technologies, the protocol, domain name, and path in the URL are converted to lowercase letters, and the default port (such as 80 for HTTP or 443 for HTTPS), redundant paths (such as paths marked by ". / " or ".. / "), default file names (such as index.html), and fragment identifiers (such as the # part) in the URL are removed. This technology can solve the problem of inconsistent URL formats, and can also normalize multiple web pages with the same page content but different URL formats, reducing the number of such duplicate web pages to a certain extent. However, there are still a large number of duplicate web pages on the Internet.

[0022] To this end, an embodiment of the present disclosure proposes a method for generating a page identifier by identifying query parameters in the URL that have no effect on page content changes and query parameters that change with the user, and using the URL after removing irrelevant query parameters as an identifier that uniquely identifies the page.

[0023] In this way, after removing irrelevant query parameters from a large number of duplicate web pages, URLs with paths and valid query parameters can be obtained, and different forms of URLs pointing to the same page content can be processed for data consistency, and duplicate URLs can be eliminated. In this way, the number of duplicate web pages is greatly reduced. On the one hand, it helps to reduce redundant data when indexing web pages and improve the accuracy of retrieval rankings. On the other hand, it simplifies the data analysis and statistical processing process and improves processing efficiency. On the other hand, it avoids repeated caching of the same resources, saves storage and bandwidth, and finally, it helps to detect and prevent duplicate or forged URL attacks and improve system security.

[0024] Figure 1 1 is a schematic diagram of an example environment 100 in which some embodiments of the present disclosure may be implemented. Figure 1 As shown, the environment 100 includes a computing unit 102, which may be any device with computing or processing capabilities. For example, the computing unit 102 may include but is not limited to a mobile phone, a tablet computer, a desktop computer, a server, etc. The computing unit 102 may execute a method for generating a page identifier. Figure 1 As shown, the environment 100 also includes page collection results 104, which are usually collected and obtained using crawler tools and other technologies under the legal authorization of the user. The collection results may include log files, which record page addresses, page contents, etc.

[0025] like Figure 1As shown, the computing unit 102 can group the page collection results 104. For example, the page collection results 104 include five page results of URL11, URL12, URL13, URL14, and URL15, and the computing unit 102 divides them into two groups 106-1 and 106-2, one group includes two page results of URL11 and URL13, and one group includes three page results of URL12, URL14, and URL15. In some embodiments, the page collection results 104 are all from the same third-party application, such as Mini Program A, and can be grouped based on the path, and the pages belonging to the same path can be grouped together. By comparing the path with the path table, the path code corresponding to the path can be obtained, and the matched path code is used as the group ID. The path table is pre-configured, and each path is configured with a different path code. In some implementations, the page collection results 104 come from different third-party applications, such as Mini Program A, Mini Program B, etc., and can be grouped based on the third-party application ID and path. The third-party application ID is compared with the application table to obtain the application code corresponding to the application ID, and the path is compared with the path table to obtain the path code corresponding to the path. The matched application code and path code are combined as the group ID. The path table and application table are pre-configured, each path is configured with a different path code, and each third-party application ID is configured with a different application code.

[0026] like Figure 1As shown, the calculation unit 102 processes the page results of the two groups 106-1 and 106-2. When the calculation unit 102 determines that the page contents of the two pages of URL11 and URL13 in the first group 106-1 are highly similar, it can be determined that the two pages are duplicate pages, and finally the unique identifier of the page is determined to be page identifier 1 at 108. In some embodiments, the page contents corresponding to the complete URL11 and the complete URL13 are compared, and the document object model (DOM) is used to understand the structure and content of the page before loading is completed, and the DOM original text is processed using the Embedding vector technology to obtain two page vectors of the page contents corresponding to the complete URL11 and the complete URL13. The similarity of the page contents corresponding to the complete URL11 and the complete URL13 can be calculated using the distance formula or the vector variance formula. If the similarity condition is met, it is determined that the page contents corresponding to the complete URL11 and the complete URL13 are the same, and it is necessary to remove redundant URLs. If URL11 is retained, URL11 is page identification 1, and if URL13 is retained, URL13 is page identification 1. In some embodiments, the query parameters of URL11 and URL13 and other information thereafter are removed, and both URL11 and URL13 are incomplete URLs containing paths. The page contents corresponding to the URL11 containing the path and the URL13 containing the path are compared, and the DOM original text is processed using the Embedding vector technology to obtain two page vectors of the page contents corresponding to the URL11 containing the path and the URL13 containing the path. By using the distance formula or the vector variance formula, the similarity of the page contents corresponding to the URL11 containing the path and the URL13 containing the path can be calculated. If the similarity condition is met, it is determined that the page contents corresponding to the URL11 containing the path and the URL13 containing the path are the same, and it is necessary to remove redundant URLs, such as using the URL11 or URL13 path as page identification 1. Compared with the former implementation, this implementation has less data processing and higher processing efficiency. In addition, in the above two implementations, when the page collection results come from different third-party applications, the former implementation can determine the page identifier 1 based on the third-party application ID and any complete URL. The latter implementation can determine the page identifier 1 based on the third-party application ID and the path. The formation process of the page identifier 1 can refer to the formation process of the group ID in the previous text.

[0027] like Figure 1As shown, the calculation unit 102 determines the similarity of the page contents of the three pages of URL12, URL14 and URL15 in the first group 106-2. In one example, the page contents of the above three pages can use DOM to understand the structure and content of the page before loading is completed, and use the Embedding vector technology to process the DOM original text to obtain the page vectors of the page contents corresponding to the three different URLs. The vector variance formula is used to calculate the vector variance to determine the similarity of the page contents of the three pages. Since the page content of the smart phone in URL12 is a smart phone, it is quite different from the page content of the non-smart phone with a cover in URL14 and URL15, so it is considered that there are differences in the content of this group. Then, the calculation unit identifies the invalid parameters in URL12, URL14 and URL15 respectively, that is, the target part 110-1 of URL12, the target part 110-2 of URL14 and the target part 110-3 of URL15 are identified. The target parts 110-1, 110-2, and 110-3 respectively include at least one of a first query parameter related to the user and a second query parameter unrelated to the content. The first query parameter related to the user refers to a query parameter that changes with the user, such as the user ID uid, the order ID order_id, etc. The second query parameter unrelated to the page content refers to a query parameter that does not change with the page content, such as the product ID, date, etc.

[0028] Next, the calculation unit 102 can obtain URL12 without URL12 target part 110-1, URL14 without URL14 target part 110-2, and URL15 without URL15 target part 110-3 based on URL12 target part 110-1, URL14 target part 110-2, and URL15 target part 110-3, respectively. Afterwards, a group ID is formed according to the path and the effective query parameters, or a group ID is formed according to the third-party application ID, the path, and the effective query parameters. After grouping, URL12 without URL12 target part 110-1 is a group, and URL14 without URL14 target part 110-2 and URL15 without URL15 target part 110-3 are a group. The calculation unit 102 can determine the identifier of the page of URL12 as page identifier 2 at 112-1 based on URL12 target part 110-1. After determining that the page contents of URL14 without the target portion 110-2 of URL14 and URL15 without the target portion 110-3 of URL15 are similar, the calculation unit 102 normalizes the two pages of URL14 and URL15, and determines the page identifier of URL14 or URL15 as page identifier 2 at 112-2. The following will take page identifier 2 as an example to illustrate the process of determining the page identifier. In some embodiments, the page collection results come from the same third-party application, and URL12 without the target portion of URL12 is used as page identifier 2. For example, by comparing the path table and the query parameter table, the path code and the parameter code in URL12 without the target portion of URL12 are determined, and the path code and the parameter code are combined into page identifier 2. In some embodiments, the page collection results come from multiple third-party applications, and URL12 without the target portion of URL12 and the third-party application ID are used as page identifier 2. For example, by comparing the path table and the query parameter table, the path code and the parameter code in the URL 12 without the target part of the URL 12 are determined, and the application code corresponding to the third-party application ID is determined, and the application code, the path code and the parameter code are combined into the page identifier 2. The query parameter table is pre-configured, and each query parameter is configured with a different parameter code.

[0029] In this way, the computing unit can normalize a large number of web pages. On the one hand, it can identify duplicate pages with the same page content, remove duplicate URLs, and generate page identifiers that identify unique pages. On the other hand, it can perform more fine-grained identification of multiple pages with different page contents. By identifying and removing invalid query parameters in the URL, a URL with a path and valid query parameters can be obtained. In this way, a simplified form of URL can be used to identify duplicate pages, with high processing efficiency and high deduplication rate.

[0030] It should be understood that the architecture and functions in the example environment 100 are presented for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0031] Figure 2 FIG. 2 is a flow chart of a method 200 for detecting a switch state of a confined space according to some embodiments of the present disclosure. The method 200 may be, for example, Figure 1 The computing unit 102 in the environment 100 shown is executed. Figure 2 As shown, in block 202, method 200 may obtain page data, the page data including page address and page content. In some embodiments, the page data of the same third-party application with legal authorization is obtained through a page crawler. In other embodiments, the page data of multiple third-party applications with legal authorization is obtained through a page crawler.

[0032] In box 204, method 200 may include determining a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to the user and a second query parameter unrelated to the page content. The target portion refers to invalid query parameters in the page address, mainly query parameters that change with the user and query parameters that do not change with the page content. In some embodiments, a page address may determine the first query parameter and the second query parameter, or determine any one of the first query parameter and the second query parameter. Since a page address may have multiple query parameters, the determined first query parameter may be one or more, and the second query parameter may be one or more.

[0033] In box 206, method 200 may include generating a page identifier based on the target portion. In some embodiments, the page data is collected from the same third-party application, and the target portion in the page address is removed to obtain the page address with invalid query parameters removed. Based on the page address with invalid query parameters removed, a page identifier is generated. In other embodiments, the page data is collected from multiple different third-party applications, that is, there are multiple different third-party application IDs, and the target portion in the page address is removed to obtain the page address with invalid query parameters removed. Based on the third-party application ID and the page address with invalid query parameters removed, a page identifier is generated.

[0034] In this way, invalid query parameters in the page address can be removed and the form of the page address can be simplified. When the pages corresponding to the simplified page address are the same, the simplified page address can be used as the unique identifier of the page, and based on the page identifier, duplicate pages can be removed to ensure that one page identifier corresponds to one page. Regardless of web page indexing, data analysis and statistics, data caching, or security protection identification application scenarios, page understanding can be performed based on page data with a high deduplication rate, with high processing efficiency and accuracy.

[0035] Figure 3 A flowchart of a process 300 for determining a first query parameter and a second query parameter according to some embodiments of the present disclosure is shown. Box 302 in the process 300 can obtain page data and perform unique visitor value (VC) statistics on the page data, and can count the number of times a user visits each URL, i.e., the unique visitor value of URL visits, and count the number of times a user visits each query parameter, i.e., the unique visitor value of query parameter visits. For example, user ID1 visits URL1, and user ID2 visits URL1 and URL2. The unique visitor value of URL1 is 2, and the unique visitor value of URL2 is 1.

[0036] At block 304, it is determined whether the unique visitor value is equal to 1. At block 306, in response to the unique visitor value being equal to 1, a query parameter related to the user in the page address is determined as the first query parameter in. In some embodiments, in response to the unique visitor value of the page address being equal to 1 and the unique visitor value of the query parameter being equal to 1, a query parameter in the page address that varies with the user is determined as the first query parameter. At blocks 310, 312, and 314, based on the page content, a query parameter in the page address that is not related to the page content is determined as the second query parameter.

[0037] In some embodiments, in box 310, based on the path and all query parameters in the page address, multiple page sub-addresses corresponding to the page content are formed, and each page sub-address contains the path in the page address and a query parameter. At box 312, the similarity of the page content corresponding to the multiple page sub-addresses is determined. If the similarity of the multiple page contents belonging to the same page sub-address is greater than the first threshold, it is considered that the page content has changed while the page sub-address remains unchanged, that is, the query parameter in the page sub-address has not changed, and the query parameter is confirmed as a query parameter (i.e., the second query parameter) that is unrelated to the page content at box 314. In some examples, based on multiple page sub-addresses, multiple page vectors corresponding to the page sub-addresses are extracted. For example, the page vector of the DOM original text corresponding to the page sub-address is extracted using embedding technology. Then, based on the multiple page vectors corresponding to the page sub-address, the vector variance of the multiple pages belonging to the same page sub-address is determined, and the vector variance characterizes the similarity of the page content corresponding to the multiple page sub-addresses. In response to a vector variance of a plurality of pages belonging to the same page sub-address being greater than a first threshold, a query parameter in the same page sub-address corresponding to different page contents is determined as a second query parameter.

[0038] In some embodiments, in box 310, based on the path and all query parameters in the page address, multiple page sub-addresses corresponding to the page content under the same third-party application ID are formed, and each page sub-address contains the path in the page address and a query parameter. At box 312, based on the multiple page sub-addresses, multiple page vectors corresponding to the page sub-addresses are extracted. Based on the multiple page vectors corresponding to the page sub-addresses, the vector variance of the multiple pages belonging to the same third-party application ID and the same page sub-address is determined, and the vector variance characterizes the similarity of the page content corresponding to the multiple page sub-addresses. In box 314, in response to the vector variance of the multiple pages belonging to the same page sub-address being greater than the first threshold, the query parameter in the same page sub-address corresponding to the different page content under the same third-party application ID is determined as the second query parameter.

[0039] In some embodiments, the first query parameter and the second query parameter may be obtained by processing in parallel, such as Figure 3The process is shown in the solid line. In some embodiments, the first query parameter and the second query parameter can be determined successively. In one example, in box 304, in response to the independent visitor value not being equal to 1, the process enters box 310 through the dotted arrow, that is, after the step of determining whether the page address has the first query parameter, it is determined whether the page sub-address has the second query parameter. In another example, after determining the first query parameter in box 306, the process enters box 316 through the dotted arrow to process the page address, and obtains the page address after the first query parameter is removed. After that, the process enters box 318 along the dotted arrow, and based on the page address after the first query parameter is removed, a plurality of page sub-addresses are formed, each of which contains a query parameter. Finally, the process enters box 312 along the dotted arrow, and it is determined that the similarity of the plurality of page contents belonging to the same page sub-address is greater than the first threshold value, and the second query parameter is determined in box 314. In this embodiment, the page address form when determining the second query parameter is relatively simple, and the formed page sub-addresses are relatively few, so the similarity of the page content can be determined quickly, and the second query parameter can be determined.

[0040] Regardless of the parallel implementation or the sequential implementation, when determining whether the page content similarity is greater than the first threshold, multiple page sub-addresses need to be grouped. The grouping principle is based on the group ID formed by the path and query parameters, or the group ID formed by the third-party application ID, path and query parameters. DOM understanding and vector extraction are performed on the same group. When the page content of the same group is greater than the first threshold, the query parameter corresponding to the group can be determined to be the second query parameter. The first threshold can be preset, such as 0.001.

[0041] Figure 4 FIG. 4 is a schematic diagram showing an example process 400 for determining a first query parameter and a second query parameter according to some embodiments of the present disclosure. Figure 4 As shown, the process 400 can obtain sample page data 402, which includes the page addresses of two pages (the two URLs use the same protocol and domain name, which are omitted in the figure) and page content (only simply shown in text, actually including page title, page text information, page image information, etc.). The first page is the page address and page content for placing an order for a shared bicycle, and the second page is the page address and page content for returning a shared bicycle. As can be seen in the figure, the paths in the URLs of the two pages are the same, so the two pages are grouped in the same group. Afterwards, according to Figure 3 The illustrated process 300 performs target part identification and normalization. Figure 3 Take parallel processing flow as an example implementation. Figure 3 Block 310 may be executed in parallel with block 304 or after block 304. Figure 4Implemented before determining the unique visitor value. Figure 4 , the example page data 402 is processed into a page data table 404, which includes multiple page sub-addresses and their page content, UV, and content changes. Among them, there are actually two first page sub-addresses, but the two are the same, so the records are merged in the table (it can also be understood as merging the same group of information), and the two page sub-addresses correspond to the order page content and the return page content respectively. Similarly, the second page sub-address is the same. Among them, the UV counted in the page data table 404 is the unique visitor value of the query parameter.

[0042] like Figure 4 , the content change in the page data packet 404 is judged based on the right process. Taking the first page sub-address as an example, it has two page sub-addresses 420-1 and 420-2 corresponding to different page contents. Through DOM, the structure and content of each page sub-address before loading is completed are understood, and the DOM original text is processed using the Embedding vector technology to obtain two page vectors of the page content corresponding to each page sub-address, that is, the first vector and the second vector are obtained after processing by box 422. Box 424 calculates the vector variance of the first vector and the second vector. When the vector variance is greater than the first threshold, it is determined that the contents of the two pages with the same page sub-address are different. Box 416 outputs the content change and records it in the data table. In addition, the second page sub-address is also calculated in the same way as the above example to determine that the content has changed. For the third, fourth, fifth, and sixth page sub-addresses, since each group has only one page content, it is recorded as Null.

[0043] like Figure 4 , based on the page data table 404, the process of box 406 and box 408 can be executed. In box 406, based on UV=1, the query parameters in the second, fifth, and sixth page sub-addresses can be determined to be the first query parameters. In box 408, based on the change of content and the unchanged query parameters, the query parameters in the first and second page sub-addresses can be determined to be the second query parameters. After normalization processing in box 410, a normalized result is obtained, that is, the page sub-address after the first and second query parameters are removed, leaving the third and fourth page sub-addresses. In other words, the original URL in the example page data 402 is processed into the third and fourth page sub-addresses in box 402. Correspondingly, the page identifier Page ID 1 and the page identifier Page ID 2 are determined based on the third and fourth page sub-addresses, respectively. In addition, if the page data comes from multiple third-party applications, the page identifier Page ID 1 can also be determined based on the third-party application ID and the third page sub-address, and the page table identifier Page ID 2 can also be determined based on the third-party application ID and the fourth page sub-address.

[0044] In some implementations, after determining the first query parameter and the second query parameter, they may also be stored in the invalid query parameter list 412. In this way, the invalid query parameter list 412 that collects a large number of invalid query parameters may be used to directly remove invalid query parameters from the URL, forming a relatively simple URL, and quickly identifying multiple duplicate pages.

[0045] Figure 5 FIG. 5 shows a flowchart of a page content understanding process 500 according to some embodiments of the present disclosure. Figure 5 , in box 502, the process 500 can understand the page content. In box 504, under the legal authorization of the user, the page is sampled to obtain page data. In some embodiments, the obtained page data is grouped, and each group is grouped by a group ID formed based on a path, or a group ID formed based on a third-party application ID and a path. In box 506, the page content similarity is judged for multiple pages in the same group. In some examples, in response to the vector variance of the page content of multiple pages being less than the second threshold, it is considered that the multiple page contents in the same group are the same, and the page identifier Page DI=app_id+path is determined, where app_id is the third-party application ID and path is the path. If the page data comes from the same third-party application, the page identifier is Page DI=path. The page identifier can be formed based on the original parameters or converted into respective codes. In some examples, in response to the vector variance of the page content of multiple pages being not less than the second threshold, it is considered that there are differences in the content of multiple pages in the same group, and then the next finer granularity is entered for identification. Among them, the second threshold is a pre-set threshold, which can be 0.001.

[0046] like Figure 5 In block 512, it is determined whether each page address in the same group contains the first query parameter or the second query parameter. In some embodiments, each query parameter in the page address is queried in the invalid query parameter list. If a match is found, the invalid query parameter can be identified and the subsequent block 514 process can be entered. In some embodiments, according to Figure 3 The example process shown identifies the first query parameter and the second query parameter. Then, the process proceeds to the subsequent block 514. In some embodiments, the invalid query parameter list may be used to first identify a round of invalid query parameters, and then the page address after removing the invalid query parameters identified in the first round may be further identified according to the invalid query parameter list. Figure 3 The illustrated example process identifies a first query parameter and a second query parameter.

[0047] In some embodiments, after identifying that the page address contains at least one of the first query parameter and the second query parameter, in box 513, grouping can be performed based on the third-party application ID, path, and valid query parameter, or based on the path and valid query parameter. The page content similarity is calculated for the grouped page sub-addresses. If the variance vector of the page content is less than the third threshold 516, the page content of multiple pages in the same group is similar, and the group ID can be determined as the page identifier. For example, the page identifier Page DI = app_id + path + valid query parameter. For another example, the page identifier Page DI = path + valid query parameter. Among them, the third threshold is a pre-set threshold, which can be 0.001. In some examples, the first, second, and third thresholds can also be set to other values ​​as needed, and the three can be the same, or they can be different in pairs or different from each other. In some embodiments, in response to the page content similarity being not less than the third threshold, it can return to 512, or enter box 518 to determine the page identifier based on the complete URL.

[0048] In some embodiments, if the page address does not contain the first query parameter or the second query parameter, in block 518, a page identifier is formed based on each page address. After the process passes through blocks 506 and 512, it can be determined that the page content of the page corresponds to one page address, that is, the page content does not have multiple URLs. For example, the page identifier Page DI = app_id + full URL. For another example, the page identifier Page DI = full URL.

[0049] In some implementations, after understanding at different granularities, the process proceeds to block 510 to form a page understanding dimension table, in which duplicate pages are substantially removed, to form page data in which one page identifier corresponds to one page content.

[0050] In this way, page understanding is performed at different granularities. Normalized URLs can greatly improve data consistency, processing efficiency, and system security.

[0051] Figure 6 A block diagram of an apparatus 600 for generating a page identifier according to some embodiments of the present disclosure is shown. The apparatus 600 includes an acquisition module 602 configured to acquire page data, the page data including a page address and page content. The apparatus 600 also includes a first determination module 604 configured to determine a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to the user and a second query parameter unrelated to the page content. The apparatus 600 also includes a generation module 606 configured to generate a page identifier based on the target portion.

[0052] In some embodiments, the apparatus 600 further includes a second determination module configured to determine the similarity of the page contents of the multiple pages belonging to the same path. The generation module 606 includes a first generation unit configured to generate a page identifier identifying the multiple pages based on the path in response to the similarity of the page contents of the multiple pages belonging to the same path being not greater than a second threshold. The generation module 606 further includes a second generation unit configured to generate multiple page identifiers identifying the multiple pages based on the multiple page addresses with the target portion removed from each of the multiple pages in response to the similarity of the page contents of the multiple pages belonging to the same path being greater than a second threshold.

[0053] In some embodiments, the generation module 606 also includes a third generation unit, which is configured to generate multiple page identifiers identifying multiple pages based on the page address in response to the similarity of the page content of multiple pages belonging to the same path being greater than a second threshold and the page address of the page not having the first query parameter and the second query parameter.

[0054] Figure 7 Schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. Figure 7 As shown, the device 700 includes a processor 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 and loaded into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0055] The various processes and processing described above, such as method 200, may be performed by processor 701. For example, in some embodiments, method 200 may be implemented as a software program that is tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program may be loaded and / or installed on device 700 via ROM 702. When the software program is loaded into RAM 703 and executed by processor 701, one or more actions of method 200 described above may be performed.

[0056] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0057] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0058] The present disclosure may be a method, an apparatus, a system and / or a program product. The program product may include a machine-readable storage medium on which are loaded machine-readable program instructions for executing various aspects of the present disclosure. The machine-readable program instructions described herein may be downloaded from the machine-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or a network interface in each computing / processing device receives a machine-readable program instruction from the network, and forwards the machine-readable program instruction for storage in a machine-readable storage medium in each computing / processing device.

[0059] The machine program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The machine-readable program instructions may be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the machine-readable program instructions, and the electronic circuit may execute the machine-readable program instructions, thereby realizing various aspects of the present disclosure.

[0060] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0061] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating a page identifier, comprising: Obtaining page data, wherein the page data includes a page address and page content; Based on the page data, determining a target portion of the page address, the target portion comprising at least one of a first query parameter related to the user and a second query parameter unrelated to the page content; as well as Based on the target portion, a page identifier is generated.

2. The method according to claim 1, wherein the page data further includes a unique visitor value, and based on the page data, determining the target portion of the page address comprises: In response to the unique visitor value being equal to 1, determining a query parameter related to the user in the page address as the first query parameter; as well as Based on the page content, a query parameter in the page address that is unrelated to the page content is determined as the second query parameter.

3. The method according to claim 2, wherein in response to the unique visitor value being equal to 1, determining the query parameter related to the user in the page address as the first query parameter comprises: In response to the unique visitor value of the page address being equal to 1 and the unique visitor value of the query parameter being equal to 1, the query parameter in the page address that changes with the user is determined as the first query parameter.

4. The method according to claim 2, wherein based on the page content, determining the query parameter in the page address that is unrelated to the page content as the second query parameter comprises: Based on the path and all query parameters in the page address, forming a plurality of page sub-addresses corresponding to the page content, each of the page sub-addresses including the path and a query parameter in the page address; Based on the plurality of the page sub-addresses, extracting a plurality of page vectors corresponding to the page sub-addresses; Determine, based on the plurality of page vectors corresponding to the page sub-addresses, a vector variance of the plurality of pages belonging to the same page sub-address, wherein the vector variance represents a similarity of the page contents corresponding to the plurality of page sub-addresses; as well as In response to the vector variance of the plurality of pages belonging to the same page sub-address being greater than a first threshold, a query parameter in the same page sub-address corresponding to different page contents is determined as the second query parameter.

5. The method according to claim 2, wherein the page data further includes an application ID, and based on the page content, determining the query parameter in the page address that is unrelated to the page content as the second query parameter comprises: Based on the path and all query parameters in the page address, forming a plurality of page sub-addresses corresponding to the page content under the same third-party application ID, each of the page sub-addresses including the path in the page address and a query parameter; Based on the plurality of the page sub-addresses, extracting a plurality of page vectors corresponding to the page sub-addresses; Based on the plurality of page vectors corresponding to the page sub-addresses, determining a vector variance of the plurality of pages belonging to the same third-party application ID and the same page sub-address, wherein the vector variance represents a similarity of the page contents corresponding to the plurality of page sub-addresses; as well as In response to the vector variance of the plurality of pages belonging to the same page sub-address being greater than a first threshold, a query parameter in the same page sub-address corresponding to different page contents under the same third-party application ID is determined as the second query parameter.

6. The method according to any one of claims 1 to 5, further comprising: In response to the similarity of the page contents of the plurality of pages belonging to the same path being not greater than a second threshold, a page identifier identifying the plurality of pages is generated based on the path.

7. The method according to claim 6, wherein in response to the similarity of the page contents of the plurality of pages belonging to the same path being not greater than a second threshold, generating a page identifier for identifying the plurality of pages based on the path comprises: Based on the paths in the plurality of page addresses, extracting a plurality of page vectors corresponding to the paths; Determining, based on the plurality of page vectors corresponding to the paths, vector variances of the plurality of pages belonging to the same path, the vector variances representing similarities of the page contents; as well as In response to the vector variance of the plurality of pages belonging to the same path being no greater than a second threshold, a page identifier identifying the plurality of pages is generated.

8. The method according to claim 6, further comprising: In response to the similarity of the page contents of the plurality of pages belonging to the same path being greater than the second threshold, a plurality of page identifiers identifying the plurality of pages are generated based on the plurality of page addresses each with the target portion removed.

9. The method according to claim 8, wherein the page data further includes a third-party application ID, and based on the path, generating a page identifier for identifying the plurality of pages further includes: Based on the third-party application ID and the path, generate a page identifier that identifies the plurality of pages; Generating a plurality of page identifiers identifying a plurality of the pages based on the plurality of page addresses each having the target portion removed comprises: Based on the third-party application ID and the plurality of page addresses with the target portion removed from each of the plurality of pages, a plurality of page identifiers identifying the plurality of pages are generated.

10. The method according to claim 9, wherein generating a plurality of page identifiers identifying a plurality of the pages based on the plurality of page addresses from which the target portion is removed, further comprises: In response to the similarity of the page contents corresponding to the multiple page addresses from which the target portions are respectively removed being not greater than a third threshold, multiple page identifiers identifying the multiple pages are generated based on the multiple page addresses from which the target portions are respectively removed.

11. The method according to claim 8, wherein in response to the similarity of the page contents of the plurality of pages belonging to the same path being greater than the second threshold, the method further comprises: The target portion in the plurality of page addresses is determined based on an invalid query parameter list, wherein the invalid query parameter list stores the first query parameter and the second query parameter.

12. The method according to claim 8, further comprising: In response to the similarity of the page contents of the plurality of pages belonging to the same path being greater than the second threshold and the page addresses of the pages not having the first query parameter and the second query parameter, a plurality of page identifiers identifying the plurality of pages are generated based on the page addresses.

13. The method according to claim 12, wherein the page data further includes a third-party application ID, and generating a plurality of page identifiers for identifying the plurality of pages based on the plurality of page addresses with the target portion removed from each of the plurality of page addresses further includes: Based on the third-party application ID and the plurality of page addresses, a plurality of page identifiers identifying the plurality of pages are generated.

14. The method according to any one of claims 1 to 5, further comprising: The first query parameter and the second query parameter are stored in an invalid query parameter list.

15. A device for generating a page identifier, comprising: An acquisition module is configured to acquire page data, wherein the page data includes a page address and page content; A first determination module is configured to determine a target portion of the page address based on the page data, wherein the target portion includes at least one of a first query parameter related to the user and a second query parameter unrelated to the page content; as well as The generating module is configured to generate a page identifier based on the target portion.

16. The apparatus according to claim 15, further comprising: A second determination module is configured to determine the similarity of the page contents of the plurality of pages belonging to the same path; The generation module comprises: A first generating unit is configured to generate a page identifier for identifying the plurality of pages based on the path in response to the similarity of the page contents of the plurality of pages belonging to the same path being not greater than a second threshold; The second generating unit is configured to generate a plurality of page identifiers identifying the plurality of pages based on the plurality of page addresses each with the target portion removed in response to the similarity of the page contents of the plurality of pages belonging to the same path being greater than the second threshold.

17. The apparatus according to claim 16, wherein the generating module further comprises: The third generating unit is configured to generate a plurality of page identifiers identifying the plurality of pages based on the page addresses in response to the similarity of the page contents of the plurality of pages belonging to the same path being greater than the second threshold and the page addresses of the pages not having the first query parameter and the second query parameter.

18. Electronic equipment, including processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 14.

19. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images