A method for evaluating webpage importance based on local search

Abstract

This invention implements a webpage importance assessment method based on local search using methods in the field of network technology. Step S1: Transform webpages and their links into a graph structure; Step S2: Sample a preset number of random walks on the graph structure; Step S3: Perform a local search on the target node t to obtain the final parameters of the local search; Step S4: Calculate an estimated Page Ranking Score for the target node t based on the terminating node of the random walk sampling in Step S2, the number of samplings, and the final parameters of the local search in Step S3; Step S5: Determine if the estimated value is sufficiently accurate; Step S6: Evaluate the importance of the webpage corresponding to node t based on the estimated Page Ranking Score of the target node t. This solution can assess the importance of a target webpage by performing a local search operation on the Internet, without relying entirely on a global random walk on the Internet for assessment.

Description

Technical Field

[0001] This invention relates to webpage ranking in the field of computer technology, and in particular to a method for evaluating the importance of webpages based on local search. Background Technology

[0002] In the era of big data, with the exponential growth of internet information and the increasing demands of users for information quality, assessing the importance of web pages has become a crucial task in the field of information retrieval. This process is not only vital to search engine result ranking but also directly impacts user experience and the efficiency of information retrieval. Determining the value and importance of specific web pages on the internet has become a core issue of great concern to numerous entities, including search engine companies, universities, and research institutions. Among various methods, search engines, led by Google and Baidu, widely use PageRank to measure the importance of web pages, aiming to calculate the PageRank score of a specific web page on the internet. However, given the massive amount of information stored on the current internet, the existing PageRank calculation method is insufficient in terms of operational efficiency and accuracy, making it difficult to adapt to the needs of ultra-large-scale internet data. In particular, the ranking calculation method used in the scheme disclosed in patent CN116992153A, although it has certain performance advantages, is only applicable to the case where the link relationship between web pages is modeled as an undirected relationship. Furthermore, it does not consider the combination of local search and random walk sampling, so it cannot be applied to the more general case where the link relationship between web pages is modeled as a directed relationship, and its computational performance is not the optimal solution. Summary of the Invention

[0003] To address the aforementioned problems, the purpose of this invention is to provide a method for determining the importance of a webpage, which, by performing a local search on the target webpage, can calculate the Page ranking score of a specified webpage in a time-efficient manner compared to existing methods.

[0004] To achieve the above objectives, the present invention adopts the following technical solution, including steps S1-S6:

[0005] Step S1: Transform the web pages and the links between them into a graph structure;

[0006] Step S2: Sample a preset number of random walks on the graph structure to obtain the termination node and sampling number parameters for each random walk.

[0007] Step S3: Perform a local search on the target node t until the time spent on the local search is about to exceed the time spent on step S2, and obtain the final parameters of the local search.

[0008] Step S4: Based on the terminal node and sampling count of the random walk in Step S2, and the final parameters of the local search in Step S3, calculate the estimated Page rank score of the target node t.

[0009] Step S5: Based on the sampling number of the random walk in step S2, the final parameters of the local search in step S3, and the estimated values ​​calculated in step S4... Determine whether the estimated value is accurate enough. If it is not accurate enough, double the number of random walks in step S2 and repeat steps S2-S4.

[0010] Step S6: Based on the estimated ranking score of the target node t Page Evaluate the importance of the webpage corresponding to node t.

[0011] The graph structure is represented as G, which includes several nodes that correspond one-to-one with the webpages and multiple directed edges that correspond to the relationships between the webpages. The target webpage corresponds to the target node t in the graph structure.

[0012] A webpage refers to all webpages within a certain scope, and the relationships between webpages refer to the links between them. In the process of converting webpages into a graph structure, all webpages on the search engine are mapped to graph nodes, and the links between webpages are mapped to edges in the graph structure. If webpage u has a link that leads to webpage v, then a directed edge (denoted as edge (u,v)) is created on the graph structure pointing from node u to node v. The set of vertices in this graph structure G is denoted as V, and the number of vertices and edges in graph structure G are denoted as n and m, respectively. The node pointed to by node u is called the out-neighbor node of node u, and the set of all out-neighbor nodes of node u is called the out-neighbor set N of node u. out (u); The node pointing to node u is called the incoming neighbor node of node u, and the set of all incoming neighbor nodes of node u is called the incoming neighbor set N of node u. in (u), and the number of outgoing neighbor nodes of node u in the graph structure is called the outgoing degree d of node u. out (u), where the number of in-neighbor nodes of node u in the graph structure is called the in-degree d of node u. in (u).

[0013] In the predetermined number of random walks, the predetermined number is n. r =100.

[0014] Each sampling random walk process includes the following steps:

[0015] Step S201: Randomly select a node from the entire graph structure as the current node;

[0016] Step S202: Generate a uniform random number r in the range [0,1]. If r is less than or equal to α, then the current node is taken as the termination node of the random walk. Otherwise, proceed to step S203, where α is the decay coefficient and its value range is (0,1).

[0017] Step S203: Randomly select an outgoing neighbor node of the current node, update the current node to the outgoing neighbor node, and repeat step S202.

[0018] The specific steps for performing a local search on t include:

[0019] Step S301: Set the initial local search parameter to ε = 1 / 2;

[0020] Step S302: Assign 1 unit of probability at the target node t; let r(v) represent the cumulative probability of node v at the current time, where r(t) = 1 at the current time;

[0021] Step S303: Arbitrarily select a node v that satisfies r(v)>ε. For each incoming neighbor u of v, according to the formula r(u)=r(u)+(1-α)r(v) / d out (u) Update the value of the probability r(u) of node u;

[0022] Step S304: Update the probability p(v) of node v according to the formula p(v)=p(v)+αr(v), where the initial value of p(v) is 0 before the start of the local search process;

[0023] Step S305: Update the probability r(v) of node v according to the formula r(v) = 0;

[0024] Step S306: Repeat steps S302-S305 until there is no node v that satisfies r(v)>ε;

[0025] In step S307, after halving the local search parameters, repeat steps S301-S306 until the time spent on the local search is about to exceed the time spent on step S2.

[0026] The estimated Page ranking score of the target node t is calculated. The method includes: step S401, calculating

[0027]

[0028] Step S402: For each terminal node v of the random walk sampled in step S2, update... The update method is as follows:

[0029]

[0030] The method for determining whether the estimated value is accurate enough is to... judge Whether it is true or false, if it is true, the judgment is accurate enough; if it is false, it is not accurate enough.

[0031] The present invention has the following advantages due to the adoption of the above technical solutions:

[0032] 1. This method uses local search operations on the target webpage on the Internet to evaluate the importance of the target webpage, without relying entirely on random walks across the entire Internet for evaluation.

[0033] 2. Using random walk as a method to limit the cost of local search and as a basis for calculating importance, the optimal calculation result of Page's ranking score for the target webpage is achieved. The dynamic adjustment process used in steps S3 and S5 is used to balance the computational cost of local search and random walk sampling, so as to obtain a sufficiently accurate estimate of Page's ranking score with the optimal total cost. Attached Figure Description

[0034] Figure 1 This is a flowchart of a method for evaluating the importance of web pages according to an embodiment of the present invention; Detailed Implementation

[0035] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0036] This invention proposes a method for evaluating the importance of web pages based on local search.

[0037] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0038] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0039] In today's context of rapid development in information technology, the internet has become a crucial platform for information acquisition and exchange. Search engines, such as Baidu and Google, as typical representatives of internet information retrieval systems, primarily help users efficiently retrieve the information they need from the internet and display the search results in webpage format. To further enhance the user search experience, mainstream search engine operators generally adopt the practice of assigning importance scores to webpages on the internet. When users perform information retrieval operations, relevant webpages with higher importance scores are prioritized to ensure high-quality search results.

[0040] In assessing webpage importance, search engines such as Baidu and Google commonly use PageRank as a metric. The core concept of PageRank is to evaluate webpage importance based on the interconnected links between webpages on the World Wide Web. Specifically, if a webpage is linked to by numerous other webpages or by authoritative websites, then that webpage is considered to have high importance. In practice, a webpage's importance can be quantified by calculating its PageRank score.

[0041] However, given the sheer volume of internet data, existing PageRank score calculation methods are significantly inefficient and struggle to meet the demands of processing massive amounts of internet data. To improve computational efficiency, current technologies typically only estimate the PageRank score of a specific webpage, rather than calculating its precise value. While this method can provide results quickly, the estimated value often deviates significantly from the actual value, failing to meet the accuracy requirements of practical applications. Therefore, this embodiment proposes an improved method that can more efficiently calculate the PageRank score of a target webpage, thereby accurately assessing its importance to meet the needs of real-world applications.

[0042] like Figure 1 The diagram shows a flowchart of a method for determining the importance of a webpage according to a preferred embodiment of the present invention, including steps S1-S6.

[0043] Step S1 involves converting web pages and their links into a graph structure. The graph structure G includes nodes corresponding one-to-one with each web page and directed edges corresponding to the relationships between the web pages. The target web page corresponds to the target node t in the graph structure. Specifically, in this embodiment, "web pages" refers to all web pages within a certain range, and "relationships" refers to the links between web pages. The following example uses all web pages on the Baidu search engine and the link network between them.

[0044] In the process of transforming web pages into a graph structure, we map all web pages on the Baidu search engine to graph nodes, and the links between web pages to edges in the graph structure. If web page u has a link that leads to web page v, then a directed edge (denoted as edge (u,v)) is created in the graph structure pointing from node u to node v. Let V be the set of vertices in the graph structure G, and let n and m be the number of vertices and edges in the graph structure G, respectively. We call the node pointed to by node u the outgoing neighbor node of node u, and the set of all outgoing neighbor nodes of node u is called the outgoing neighbor set N of node u. out (u); The node pointing to node u is called the incoming neighbor node of node u, and the set of all incoming neighbor nodes of node u is called the incoming neighbor set N of node u. in (u). Furthermore, the number of outgoing neighbors of node u in the graph structure is called the outgoing degree d of node u. out (u), where the number of in-neighbor nodes of node u in the graph structure is called the in-degree d of node u. in (u).

[0045] Step S2: Sample a certain number of random walks on the graph structure to obtain the terminal node of each random walk sampling.

[0046] Step S3: Continuously perform local searches on the target node t until the time spent on the local search is about to exceed the time spent on step S2.

[0047] Step S4: Calculate the estimated Page rank score of the target node t based on the sampling results of the random walk in Step S2 and the results of the local search in Step S3.

[0048] Step S5: Based on the sampling number of the random walk in step S2, the final parameters of the local search in step S3, and the estimated values ​​calculated in step S4... Determine if the estimated value is accurate enough. If it is not accurate enough, double the number of random walks in step S2 and repeat steps S2-S4.

[0049] Step S6: Based on the estimated ranking score of the target node t Page Evaluate the importance of the webpage corresponding to node t.

[0050] Therefore, the method for determining the importance of a webpage provided in this embodiment can more accurately estimate the Page ranking score of a specified webpage while minimizing the time required by existing methods, thereby enabling a more efficient measurement of the webpage's importance.

[0051] In one embodiment, step S2 sets the initial random walk sampling number to n. r=100, and each sampling random walk process includes the following steps S201-S203.

[0052] Step S201: Randomly select a node from the entire graph structure as the current node, that is, randomly select a webpage from all webpages on the Baidu search engine and use its corresponding node as the current node.

[0053] Step S202: Generate a uniform random number r in the range [0,1]. If r is less than or equal to α, then the current node is used as the termination node of the random walk; otherwise, proceed to step S203. Here, α is the decay coefficient, and its value ranges from (0,1).

[0054] Step S203: Randomly select an outgoing neighbor node of the current node, update the current node to the outgoing neighbor node, and repeat step S202.

[0055] In one embodiment, step S3, which involves a local search of t, specifically includes the following steps S301-S308.

[0056] Step S301: Set the initial local search parameter to ε = 1 / 2.

[0057] Step S302: Assign 1 unit of probability at the target node t; we use r(v) to represent the cumulative probability of node v at the current time. Therefore, at the current time r(t) = 1.

[0058] Step S303: Arbitrarily select a node v that satisfies r(v)>ε. For each incoming neighbor u of v, update the value of the probability r(u) of node u according to Formula 1, where Formula 1 is:

[0059] r(u)=r(u)+(1-α)r(v) / d out (u).#(1)

[0060] Step S304: Update the probability p(v) of node v according to Formula 2, where the initial value of p(v) is 0 before the start of the local search process. Formula 2 is:

[0061] p(v)=p(v)+ar(v).#(2)

[0062] Step S305: Update the probability r(v) of node v according to Formula 3, where Formula 3 is...

[0063] r(v)=0.#(3)

[0064] Step S306: Repeat steps S302-S305 until there is no node v that satisfies r(v)>ε.

[0065] In step S307, after halving the local search parameters, repeat steps S301-S306 until the time spent on the local search is about to exceed the time spent on step S2.

[0066] In one embodiment, step S4 calculates an estimate of the Page rank score of the target node t. The method includes the following steps S401-S402.

[0067] Step S401, calculate according to Formula 4 Formula 4 is

[0068]

[0069] Step S402: For each terminal node v of the random walk sampled in step S2, update according to formula 5. Formula 5 is

[0070]

[0071] In one embodiment, the estimated value is determined in step S5. The method to determine whether it is accurate enough is to judge Check if the condition is met. If it is, the condition is considered sufficiently accurate. If it is not sufficiently accurate, then the sampling number n of the random walk in step S2 is reduced. r Double the product and repeat steps S2-S4.

[0072] The method in this embodiment can achieve theoretical optimization. The probability that an estimated Page ranking score for target node t can be obtained within a certain time frame, and that the relative error between this estimated score and the accurate Page ranking score of target node t does not exceed 0.1, is at least 2 / 3. In contrast, the optimal algorithm prior to this method could only achieve this within a certain time frame. Within a given timeframe, the same objective must be achieved. Here, n is the number of nodes in the graph structure G (corresponding to the number of web pages on the internet), m is the number of edges in the graph structure G (corresponding to the number of directed links between web pages on the internet), and Δ in Let Δ be the maximum in-degree of a single node in the graph structure G (i.e., the maximum number of links on the internet pointing to a single webpage). out This represents the maximum out-degree of a single node in the graph structure G (i.e., the maximum number of links contained in a single webpage on the Internet).

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for evaluating the importance of web pages based on local search, characterized in that: Including steps S1-S6: Step S1: Transform the web pages and the links between them into a graph structure; Step S2: Sample a preset number of random walks on the graph structure to obtain the termination node and sampling number parameters for each random walk. Step S3: Perform a local search on the target node t until the time spent on the local search is about to exceed the time spent on step S2, and obtain the final parameters of the local search. Step S4: Based on the terminal node and sampling count of the random walk in Step S2, and the final parameters of the local search in Step S3, calculate the estimated Page rank score of the target node t. ; Step S5: Based on the sampling number of the random walk in step S2, the final parameters of the local search in step S3, and the estimated values ​​calculated in step S4... Determine whether the estimated value is accurate enough. If it is not accurate enough, double the number of random walks in step S2 and repeat steps S2-S4. Step S6: Based on the estimated ranking score of the target node t Page Evaluate the importance of the webpage corresponding to node t; The specific steps for performing a local search on the target node t include: Step S301, set the initial local search parameters as follows ; Step S302: Assign a probability of 1 unit at the target node t; using This represents the cumulative probability of node v at the current time. ; Step S303, arbitrarily select one that satisfies For a node v, for each incoming neighbor u of v, according to the formula The probability of updating node u The value; Step S304, the probability of node v is... According to the formula Update, among which The initial value of is 0 before the local search process begins; Step S305, the probability of node v is... According to the formula Update; Step S306: Repeat steps S302-S305 until no solution is found. node v; In step S307, after halving the local search parameters, repeat steps S301-S306 until the time spent on the local search is about to exceed the time spent on step S2.

2. The webpage importance evaluation method based on local search as described in claim 1, characterized in that: The graph structure is represented as G, which includes several nodes that correspond one-to-one with the webpages and multiple directed edges that correspond to the relationships between the webpages. The target webpage corresponds to the target node t in the graph structure. A webpage refers to all webpages within a certain scope, and the relationships between webpages refer to the links between them. In the process of converting webpages into a graph structure, all webpages on the search engine are mapped to graph nodes, and the links between webpages are mapped to edges in the graph structure. If webpage u has a link that leads to webpage v, then a directed edge is created in the graph structure from node u to node v, denoted as an edge. Let V be the set of vertices in the graph structure G, and let n and m be the number of vertices and edges in the graph structure G, respectively. The node pointed to by node u is called the out-neighbor node of node u, and the set of all out-neighbor nodes of node u is called the out-neighbor set of node u. The node that points to node u is called the incoming neighbor node of node u, and the set of all incoming neighbor nodes of node u is called the incoming neighbor set of node u. Furthermore, the number of out-neighbor nodes of node u in a graph structure is called the out-degree of node u. The number of in-neighbor nodes of node u in a graph structure is called the in-degree of node u. .

3. The webpage importance assessment method based on local search as described in claim 1, characterized in that: In the preset number of random walks, the preset number is .

4. The webpage importance evaluation method based on local search as described in claim 1, characterized in that: The random walk process for each sampling includes the following steps: Step S201: Randomly select a node from the entire graph structure as the current node; Step S202: Generate a uniform random number r in the range [0,1]. If r is less than or equal to... Then the current node is set as the termination node of the random walk; otherwise, proceed to step S203, where... is the attenuation coefficient, with a value range of (0,1); Step S203: Randomly select an outgoing neighbor node of the current node, update the current node to the outgoing neighbor node, and repeat step S202.

5. The webpage importance evaluation method based on local search as described in claim 1, characterized in that: The estimated Page ranking score of the target node t is calculated. The method includes: step S401, calculating : Step S402: For each terminal node v of the random walk sampled in step S2, update... The update method is as follows: .

6. The webpage importance evaluation method based on local search as described in claim 1, characterized in that: The method for determining whether the estimated value is accurate enough is to... judge Whether it is true or false, if it is true, the judgment is accurate enough; if it is false, it is not accurate enough.

Images