A method, device and terminal device for determining web page category based on node extraction
Through the web page category determination method based on node extraction, the similarity and PageRank value of web page content are calculated, and the weighting algorithm is combined to distinguish web page categories, which solves the problems of high resource consumption and low analysis accuracy in the existing technology, and achieves a more efficient web page Content analysis.
Patent Information
- Application Number
- CN202111570549.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-21
AI Technical Summary
In the prior art, data processing systems cannot effectively distinguish web page categories, resulting in increased resource consumption and low accuracy of content analysis.
The web page category determination method based on node extraction is adopted. By extracting multiple web page information, the content similarity and PageRank value are calculated, and combined with a preset weighting algorithm, the web page category is distinguished, resource consumption is reduced and analysis accuracy is improved.
It improves the accuracy of web page category distinction, reduces the number of times a web crawler extracts list pages, saves resources, and improves the accuracy of content analysis of main pages.
Smart Images

Figure CN114385893B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology services, and in particular to a method, device and terminal equipment for determining web page categories based on node extraction. Background Art
[0002] In an era of highly developed information technology, everyone's social network, consumption information, and movement trajectory have generated a variety of data. By collecting, integrating, analyzing and processing data, we can understand things more clearly, thereby helping us make more accurate decisions. Usually, we can organize the data we need for the problem according to our own needs, and analyze valuable information and opinions based on these data, thereby improving the efficiency of problem solving. For web page data, through the selected topic, the web page content is obtained, screened, sorted and analyzed to obtain more accurate content. Using web crawlers is a very important part of obtaining, collecting and analyzing data for web pages.
[0003] In the existing technology, in daily data collection and analysis, developers have developed "media collection" as a system for storing network media data, which can conduct data mining with a wider range and higher depth. This system is divided into: data collection, document analysis, data flow and data retrieval, but the document analysis function cannot distinguish the types of web pages, resulting in repeated parts in each analysis, increasing resource consumption; at the same time, since it cannot distinguish between list pages and text pages, the analysis accuracy of web page content is low.
[0004] Therefore, the current data processing system is in urgent need of a directory node extraction method to solve the problems of unnecessary resource consumption and low content analysis accuracy in the existing technology. Summary of the invention
[0005] The embodiments of the present invention provide a method, an apparatus and a terminal device for determining the category of a web page based on node extraction, which can improve the accuracy of distinguishing the categories of web pages.
[0006] In order to solve the above problem, an embodiment of the present invention provides a method, apparatus and terminal device for determining web page category based on node extraction, including:
[0007] Extracting a plurality of web page information, and acquiring a first content and a second content of each web page according to the plurality of web page information; wherein the first content and the second content are acquired at different time points;
[0008] Calculate the similarity between the first content and the second content in each web page respectively;
[0009] Calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page; wherein the first parameter is the parameter of the external link pointing back to itself;
[0010] According to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, the score of each web page is obtained, and according to the scores of all web pages, each web page is classified into web page categories; wherein the web page categories include list pages and text pages.
[0011] As an improvement of the above solution, extracting multiple web page information and acquiring the first content and the second content of each web page according to the multiple web page information is specifically as follows:
[0012] Extracting width data, height data and multiple node data of each web page according to the multiple web page information; wherein each node data includes position data, tag name data and text content data of a node;
[0013] Calculate the position data of the center point based on the width data and height data obtained from each web page;
[0014] Calculate the distance between the node and the center point according to the location data of the node obtained in each web page;
[0015] Determine a first node according to the distance data in each web page, and obtain the text content of each web page through the first node as the first content of each web page; after a first preset time interval, re-extract the width data, height data and node data of the multiple web pages to obtain the second content of each web page; wherein the text content includes: first text content and second text content.
[0016] As an improvement of the above solution, the first node is determined according to the distance data in each web page, and the text content of each web page is obtained through the first node, specifically:
[0017] Selecting the node with the smallest distance data in each web page as the first node;
[0018] If the tag name data of the first node is a paragraph element, then the text content data of all nodes whose tag names are paragraph elements are merged to obtain the first text content of each web page;
[0019] If the tag name of the first node is not paragraph data, the central area is determined according to the position data of the central point, and the text content data of the nodes in the central area are merged to obtain the second text content of each web page.
[0020] As an improvement of the above solution, the central area is determined according to the position data of the central point, and the text contents of the nodes in the central area are merged, specifically:
[0021] A rectangular central area is determined at the position of the central point; wherein the rectangular central area is delineated according to the mathematical golden ratio, r%=0.382, and the formula is:
[0022]
[0023] In the formula, centerX and centerY are the position data of the center point, width is the width data of the page, height is the height data of the page, and X i and Y i They are the width and height data of the center area respectively.
[0024] As an improvement to the above solution, the similarity between the first content and the second content in each web page is calculated separately, specifically:
[0025] Vectorizing the first content and the second content by using TFIDF;
[0026] The similarity between the vectorized first content and the second content is calculated to obtain a first similarity, and the calculation formula is:
[0027]
[0028] Where Sim(pi) is the similarity, C T0(pi) is the vectorization of the first content, C T(pi) Vectorization of the second content;
[0029] The first similarity is processed to obtain a second similarity, and the calculation formula is:
[0030] Sim′(p i )=1-Sim(p i )
[0031] In the formula, Sim′(p i ) is the second similarity, Sim(p i ) is the first similarity; wherein the second similarity is the similarity between the first content and the second content.
[0032] As an improvement of the above solution, the PageRank value of each first content and the PageRank value of each second content are calculated respectively to obtain the first PR value and the second PR value corresponding to each web page, specifically:
[0033] Calculate the PageRank value of the first content to obtain a third PR value, and calculate the PageRank value of the second content to obtain a fourth PR value;
[0034] The third PR value and the fourth PR value are normalized respectively to obtain a first PR value and a second PR value, wherein the normalization formula is:
[0035]
[0036] In the formula, PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, PR o (p i ) is the third PR value, PR(p i ) is the fourth PR value, max(PR) and min(PR) are the maximum and minimum values of the corresponding second PR values of all web pages; max(PR o )、min(PR o ) are the maximum and minimum values of the corresponding first PR values of all web pages.
[0037] As an improvement of the above scheme, the score of each web page is obtained according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, and the web page categories are distinguished according to the scores of all web pages, specifically:
[0038] Perform weighted calculation on the first PR value, the second PR value, the first parameter and the similarity of each web page, and obtain the fixed weight value of each dimension according to the grid search method, so as to weight each dimension and obtain the score of each page;
[0039] The scores of all pages are sorted from high to low, the first N% of the web pages are determined to be list pages, and the rest are determined to be content pages; where N is a positive number.
[0040] Correspondingly, the present invention also provides a web page category determination device based on node extraction, comprising: an information extraction module, a similarity module, a PR value calculation module and a distinction module;
[0041] The information extraction module is used to extract a plurality of web page information, and obtain a first content and a second content of each web page according to the plurality of web page information; wherein the first content and the second content are obtained at different time nodes;
[0042] As an improvement of the above solution, the information extraction module includes: a web page information extraction unit, a first position calculation unit, a second position calculation unit and a text content unit;
[0043] The web page information extraction unit is used to extract width data, height data and multiple node data of each web page according to the multiple web page information; wherein each node data includes position data, tag name data and text content data of a node;
[0044] The first position calculation unit is used to calculate the position data of the center point according to the width data and the height data obtained in each web page;
[0045] The second position calculation unit is used to calculate the distance data from the node to the center point according to the position data of the node obtained in each web page;
[0046] The text content unit is used to determine a first node according to the distance data in each web page, and obtain the text content of each web page through the first node as the first content of each web page; after a first preset time interval, re-extract the width data, height data and node data of the multiple web pages to obtain the second content of each web page; wherein the text content includes: first text content and second text content.
[0047] As an improvement of the above solution, the first node is determined according to the distance data in each web page, and the text content of each web page is obtained through the first node, specifically:
[0048] Selecting the node with the smallest distance data in each web page as the first node;
[0049] If the tag name data of the first node is a paragraph element, then the text content data of all nodes whose tag names are paragraph elements are merged to obtain the first text content of each web page;
[0050] If the tag name of the first node is not paragraph data, the central area is determined according to the position data of the central point, and the text content data of the nodes in the central area are merged to obtain the second text content of each web page.
[0051] As an improvement of the above solution, the central area is determined according to the position data of the central point, and the text contents of the nodes in the central area are merged, specifically:
[0052] A rectangular central area is determined at the position of the central point; wherein the rectangular central area is delineated according to the mathematical golden ratio, r%=0.382, and the formula is:
[0053]
[0054] In the formula, centerX and centerY are the position data of the center point, width is the width data of the page, height is the height data of the page, and Xi and Y i They are the width and height data of the center area respectively.
[0055] The similarity module is used to calculate the similarity between the first content and the second content in each web page respectively;
[0056] As an improvement of the above solution, the similarity module includes: a preprocessing unit, a first similarity unit and a second similarity unit.
[0057] The preprocessing unit is used to vectorize the first content and the second content through TFIDF;
[0058] The first similarity unit is used to calculate the similarity between the vectorized first content and the second content to obtain a first similarity, and the calculation formula is:
[0059]
[0060] Where Sim(pi) is the similarity, C T0(pi) is the vectorization of the first content, C T(pi) Vectorization of the second content;
[0061] The second similarity unit is used to process the first similarity to obtain a second similarity, and the calculation formula is:
[0062] Sim′(p i )=1-Sim(p i )
[0063] In the formula, Sim′(p i ) is the second similarity, Sim(p i ) is the first similarity; wherein the second similarity is the similarity between the first content and the second content.
[0064] The PR value calculation module is used to calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page;
[0065] As an improvement of the above solution, the PR value calculation module includes: an initial value calculation unit and a normalization unit;
[0066] The initial value calculation unit is used to calculate the PageRank value of the first content to obtain a third PR value, and calculate the PageRank value of the second content to obtain a fourth PR value;
[0067] The normalization unit is used to perform normalization processing on the third PR value and the fourth PR value respectively to obtain a first PR value and a second PR value, wherein the normalization processing formula is:
[0068]
[0069] In the formula, PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, PR o (p i ) is the third PR value, PR(p i ) is the fourth PR value, max(PR) and min(PR) are the maximum and minimum values of the corresponding second PR values of all web pages; max(PR o )、min(PR o ) are the maximum and minimum values of the corresponding first PR values of all web pages.
[0070] The differentiation module is used to obtain the score of each web page according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, and differentiate the web page categories of each web page; wherein the web page categories include list pages and text pages.
[0071] As an improvement of the above solution, the differentiation module includes: a score calculation unit and a sorting unit;
[0072] The score calculation unit is used to perform weighted calculation on the first PR value, the second PR value, the first parameter and the similarity of each web page, and obtain the fixed weight value of each dimension according to the grid search method, so as to weight each dimension and obtain the score of each page;
[0073] The ranking unit is used to rank the scores of all pages from high to low, and the first N% of the web pages are determined to be list pages, and the rest are determined to be content pages; wherein N is a positive number.
[0074] Correspondingly, the present invention also provides a computer terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, a web page category determination method based on node extraction as described in any one of the present invention is implemented.
[0075] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a web page category determination method based on node extraction as described in any one of the present invention.
[0076] As can be seen from the above, the present invention has the following beneficial effects:
[0077] The present invention provides a web page category determination method, device and terminal device based on node extraction, which extracts web page information according to a preset interval period, and respectively calculates the PR value of the web page information extracted twice, calculates the first parameter of the web page information extracted twice and calculates the similarity between the web page information extracted twice through the acquired web page information, and performs weighted calculation to obtain the score of each web page, thereby distinguishing the web page category. By distinguishing the web page category, the number of times the web crawler extracts the list page can be reduced, and the loss of resources is reduced. At the same time, the system has more memory to analyze and extract the content of the text page, thereby improving the accuracy of text extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] Figure 1 It is a flowchart of a method for determining a web page category based on node extraction provided by an embodiment of the present invention;
[0079] Figure 2 It is a structural diagram of a web page category determination device based on node extraction provided by an embodiment of the present invention;
[0080] Figure 3 It is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] Embodiment 1
[0083] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for determining a web page category based on node extraction provided by an embodiment of the present invention. Figure 1 As shown, this embodiment includes steps 101 to 104, and each step is specifically as follows:
[0084] Step 101: extracting a plurality of web page information, and acquiring a first content and a second content of each web page according to the plurality of web page information; wherein the first content and the second content are acquired at different time points.
[0085] As a preferred solution of this embodiment, multiple web page information is extracted, and the first content and the second content of each web page are obtained according to the multiple web page information, specifically: according to the multiple web page information, the width data, the height data and multiple node data of each web page are extracted; wherein each node data includes the position data, the tag name data and the text content data of a node; according to the width data and the height data obtained in each web page, the position data of the center point is calculated; according to the position data of the node obtained in each web page, the distance data from the node to the center point is calculated; according to the distance data in each web page, the first node is determined, and the text content of each web page is obtained through the first node as the first content of each web page; after a first preset time interval, the width data, the height data and the node data of the multiple web pages are re-extracted to obtain the second content of each web page; wherein the text content includes: the first text content and the second text content.
[0086] As a preferred solution of this embodiment, the first node is determined according to the distance data in each web page, and the text content of each web page is obtained through the first node, specifically: the node with the smallest distance data is selected as the first node in each web page; if the tag name data of the first node is a paragraph element, the text content data of all nodes with tag names of paragraph elements are merged to obtain the first text content of each web page; if the tag name of the first node is not paragraph data, the central area is determined according to the position data of the center point, and the text content data of the nodes in the central area are merged to obtain the second text content of each web page.
[0087] As a preferred solution of this embodiment, the central area is determined according to the position data of the central point, and the text content of the nodes in the central area is merged, specifically: a rectangular central area is determined at the position of the central point; wherein the rectangular central area is delineated according to the golden ratio of mathematics, r%=0.382, and the formula is:
[0088]
[0089] In the formula, centerX and centerY are the position data of the center point, width is the width data of the page, height is the height data of the page, and X i and Y i They are the width and height data of the center area respectively.
[0090] As a preferred solution of this embodiment, the width data, height data and multiple node data of each web page are extracted according to the information of multiple web pages, specifically: the width and height of the page or screen are determined, and the center point centerX, centerY is calculated: centerX=width / 2; centerY=height / 2;
[0091] Get all nodes D that contain content on the web page: i |i∈1,2,3,...,N}, and node d i The coordinates of the vertex (top i ,bottom i ,left i ,right i ), calculate the distance from the node to the center point i :
[0092] X i =(right i -left i ) / 2.0+left i
[0093] Yi i =(bottom i -top i ) / 2.0+top i
[0094] If satisfied: top i ≤centerX and bottom i ≥centerX, and at the same time: left i ≤centerY andright i ≥ centerY, then distance i =0;
[0095] If satisfied: top i ≤centerX and bottom i ≥centerX, then distance i =|Y i -centerY|;
[0096] If satisfied: left i ≤centerY and right i ≥ centerY, then distance i =|X i -centerX|;
[0097] If none of the above three conditions are met, then
[0098] As a preferred solution of this embodiment, a web crawler is used to capture web page content, extract all links of the web page, and distinguish nodes into directory nodes and text nodes according to whether the links exist in the directory set.
[0099] Step 102: Calculate the similarity between the first content and the second content in each web page respectively.
[0100] As a preferred solution of this embodiment, the first content and the second content are vectorized by using TFIDF; similarity is calculated between the vectorized first content and the second content to obtain a first similarity, and the calculation formula is:
[0101]
[0102] Where Sim(pi) is the similarity, C T0(pi) is the vectorization of the first content, C T(pi) is the vectorization of the second content; the first similarity is processed to obtain a second similarity, and the calculation formula is:
[0103] Sim′(p i )=1-Sim(p i )
[0104] In the formula, Sim′(p i ) is the second similarity, Sim(p i ) is the first similarity; wherein the second similarity is the similarity between the first content and the second content.
[0105] As a preferred solution of this embodiment, the first content and the second content are vectorized by TFIDF, and the calculation formula is:
[0106]
[0107]
[0108] Where W i is the word set of the main content of the web page, w ik is the word subset of the main content of the web page, T is the set of the main content of all web pages, Tw k Contains word w k The main content, Vectorize the main content of the web page.
[0109] Step 103: Calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page; wherein the first parameter is the parameter of the external link pointing back to itself.
[0110] As a preferred solution of this embodiment, the PageRank value of each first content and the PageRank value of each second content are calculated respectively. Specifically, suppose that the website W has N web pages P: i |i∈1,2,3,…,N}, where M(p i ) is all pairs of p i The set of web pages with outbound links, L(p j ) is p j The collection of all outbound links of a web page; set the PR at time t = 0 0 (p i ) with an initial value of x and a damping coefficient of α, and calculates the PageRank value PR at iteration time t t (p i ):
[0111] PR 0 (p i )=x
[0112]
[0113] Given a minimum value ∈, if the difference between the PageRank value at iteration t and the PageRank value at iteration t-1 is infinitely small, then stop iterating, otherwise continue iterating:
[0114] PR(p i )=PR t (p i ),if|PR t (p i )-PR t-1 (p i )|<∈
[0115] The PageRank value of the webpage content is obtained according to the iteration result, thereby calculating and obtaining the PageRank of the first content and the second content.
[0116] As a preferred solution of this embodiment, the PageRank value of the first content is calculated to obtain a third PR value, and the PageRank value of the second content is calculated to obtain a fourth PR value; the third PR value and the fourth PR value are normalized to obtain a first PR value and a second PR value, wherein the normalization formula is:
[0117]
[0118] In the formula, PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, PR o (p i ) is the third PR value, PR(p i ) is the fourth PR value, max(PR) and min(PR) are the maximum and minimum values of the corresponding second PR values of all web pages; max(PR o )、min(PR o ) are the maximum and minimum values of the corresponding first PR values of all web pages.
[0119] As a preferred solution of this embodiment, the first parameter of the external link node in each web page is calculated, specifically: judging whether the external link node points back to itself, the calculation formula is as follows:
[0120]
[0121] If it points back to itself, the first parameter ML(p i ) is 1, otherwise the first parameter ML(p i ) is 0.
[0122] Step 104: Based on the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, the score of each web page is obtained, and the web page category of each web page is distinguished according to the scores of all web pages; wherein the web page category includes list page and text page.
[0123] As a preferred solution of this embodiment, a weighted calculation is performed on the first PR value, the second PR value, the first parameter and the similarity of each web page, and the fixed weight value of each dimension is obtained according to the grid search method, so as to weight each dimension and obtain the score of each page; the scores of all pages are sorted from high to low, and the first N% of the web pages are determined to be list pages, and the rest are determined to be content pages; wherein N is a positive number.
[0124] As a preferred solution of this embodiment, the grid search method is used to fit the optimal coefficients of the manually annotated web page result test set and the validation set to obtain the fixed weights a, b, c, and d. The weighted calculation formula is:
[0125] Score(p i ) = aPR′(p i )+bPR′ o (p i )+cSim′(p i )+dML(p i )
[0126] So as to obtain the score of each web page Score (p i ), where PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, Sim′(p i ) is the similarity, ML(p i ) is the first parameter.
[0127] As a preferred solution of this embodiment, N can be any number between 1 and 20.
[0128] Implementing the embodiments of the present invention has the following effects:
[0129] As can be seen from the above, the present invention discloses a method for determining the category of a web page based on node extraction. This method obtains the main content of a web page by extracting node content from a central area, thereby reducing the interference of non-content data of the web page and improving the accuracy of content capture. By performing multi-dimensional calculations on the two main contents extracted from the web page, and obtaining a fixed weight value for each dimension, the fixed weight value is weighted to each dimension to obtain the score of the web page, thereby determining the web page as a list web page and a text web page. By determining the category of a web page, it is possible to avoid a web crawler from repeatedly crawling all web pages, reducing the resource consumption of obtaining the content of the text web page, and at the same time being able to focus more on the analysis of the content of the text web page, thereby improving the accuracy of the analysis of the content of the text web page.
[0130] Embodiment 2
[0131] See also Figure 2 , Figure 2 It is a structural schematic diagram of a web page category determination device based on node extraction provided by an embodiment of the present invention, including: The present invention also provides a web page category determination device based on node extraction, including: an information extraction module 201, a similarity module 202, a PR value calculation module 203 and a distinction module 204;
[0132] The information extraction module 201 is used to extract multiple web page information, and obtain the first content and the second content of each web page according to the multiple web page information; wherein the first content and the second content are obtained at different time nodes;
[0133] As an improvement of the above solution, the information extraction module 201 includes: a web page information extraction unit, a first position calculation unit, a second position calculation unit and a text content unit;
[0134] The web page information extraction unit is used to extract width data, height data and multiple node data of each web page according to the multiple web page information; wherein each node data includes position data, tag name data and text content data of a node;
[0135] The first position calculation unit is used to calculate the position data of the center point according to the width data and the height data obtained in each web page;
[0136] The second position calculation unit is used to calculate the distance data from the node to the center point according to the position data of the node obtained in each web page;
[0137] The text content unit is used to determine a first node according to the distance data in each web page, and obtain the text content of each web page through the first node as the first content of each web page; after a first preset time interval, re-extract the width data, height data and node data of the multiple web pages to obtain the second content of each web page; wherein the text content includes: first text content and second text content.
[0138] As an improvement of the above solution, the first node is determined according to the distance data in each web page, and the text content of each web page is obtained through the first node, specifically:
[0139] Selecting the node with the smallest distance data in each web page as the first node;
[0140] If the tag name data of the first node is a paragraph element, then the text content data of all nodes whose tag names are paragraph elements are merged to obtain the first text content of each web page;
[0141] If the tag name of the first node is not paragraph data, the central area is determined according to the position data of the central point, and the text content data of the nodes in the central area are merged to obtain the second text content of each web page.
[0142] As an improvement of the above solution, the central area is determined according to the position data of the central point, and the text contents of the nodes in the central area are merged, specifically:
[0143] A rectangular central area is determined at the position of the central point; wherein the rectangular central area is delineated according to the mathematical golden ratio, r%=0.382, and the formula is:
[0144]
[0145] In the formula, centerX and centerY are the position data of the center point, width is the width data of the page, height is the height data of the page, and Xi and Y i They are the width and height data of the center area respectively.
[0146] The similarity module 202 is used to calculate the similarity between the first content and the second content in each web page respectively;
[0147] As an improvement of the above solution, the similarity module 202 includes: a preprocessing unit, a first similarity unit and a second similarity unit.
[0148] The preprocessing unit is used to vectorize the first content and the second content through TF IDF;
[0149] The first similarity unit is used to calculate the similarity between the vectorized first content and the second content to obtain a first similarity, and the calculation formula is:
[0150]
[0151] Where Sim(pi) is the similarity, C T0(pi) is the vectorization of the first content, C T(pi) Vectorization of the second content;
[0152] The second similarity unit is used to process the first similarity to obtain a second similarity, and the calculation formula is:
[0153] Sim′(p i )=1-Sim(p i )
[0154] In the formula, Sim′(p i ) is the second similarity, Sim(p i ) is the first similarity; wherein the second similarity is the similarity between the first content and the second content.
[0155] The PR value calculation module 203 is used to calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page;
[0156] As an improvement of the above solution, the PR value calculation module 203 includes: an initial value calculation unit and a normalization unit;
[0157] The initial value calculation unit is used to calculate the PageRank value of the first content to obtain a third PR value, and calculate the PageRank value of the second content to obtain a fourth PR value;
[0158] The normalization unit is used to perform normalization processing on the third PR value and the fourth PR value respectively to obtain a first PR value and a second PR value, wherein the normalization processing formula is:
[0159]
[0160] In the formula, PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, PR o (p i ) is the third PR value, PR(p i ) is the fourth PR value, max(PR) and min(PR) are the maximum and minimum values of the corresponding second PR values of all web pages; max(PR o )、min(PR o ) are the maximum and minimum values of the corresponding first PR values of all web pages.
[0161] The differentiation module 204 is used to obtain the score of each web page according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, and distinguish the web page category of each web page; wherein the web page category includes list page and text page.
[0162] As an improvement of the above solution, the differentiation module 204 includes: a score calculation unit and a sorting unit;
[0163] The score calculation unit is used to perform weighted calculation on the first PR value, the second PR value, the first parameter and the similarity of each web page, and obtain the fixed weight value of each dimension according to the grid search method, so as to weight each dimension and obtain the score of each page;
[0164] The ranking unit is used to rank the scores of all pages from high to low, and the first N% of the web pages are determined to be list pages, and the rest are determined to be content pages; wherein N is a positive number.
[0165] The implementation of the embodiment of the present invention can well determine the category of a web page, the information module extracts the content of the web page, obtains the text content of the web page by determining the visual center, obtains parameters through the similarity module and the PR value calculation module, and then obtains the fixed weight value through the differentiation module to perform weighted calculation on the parameters to obtain the determination score of each web page, which is helpful to assist in improving the efficiency of web page content analysis and save resource consumption.
[0166] Embodiment 3
[0167] See also Figure 3 , Figure 3It is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention.
[0168] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the above-mentioned web page category determination method based on node extraction in the embodiment are implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of the device for determining web page category based on node extraction are shown.
[0169] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the web page category determination method based on node extraction as described in any of the above embodiments.
[0170] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.
[0171] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 301 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.
[0172] The memory 302 can be used to store the computer program and / or module. The processor 301 implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0173] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0174] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.
[0175] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for determining web page categories based on node extraction, characterized in that: include: Extracting multiple web page information, and obtaining the first content and the second content of each web page according to the multiple web page information; wherein the first content and the second content are obtained at different time nodes; extracting multiple web page information, and obtaining the first content and the second content of each web page according to the multiple web page information, specifically: extracting width data, height data and multiple node data of each web page according to the multiple web page information; wherein each node data includes position data, tag name data and text content data of a node; calculating the position data of the center point according to the width data and height data obtained in each web page; calculating the distance data from the node to the center point according to the position data of the node obtained in each web page; determining the first node according to the distance data in each web page, and obtaining the text content of each web page through the first node as the first node of each web page the first content of the web page; after a first preset time interval, re-extracting the width data, height data and node data of the plurality of web pages to obtain the second content of each web page; wherein the text content includes: the first text content and the second text content; the determining of the first node according to the distance data in each web page, and obtaining the text content of each web page through the first node, specifically: selecting the node with the smallest distance data in each web page as the first node; if the tag name data of the first node is a paragraph element, merging the text content data of all nodes with the tag name of the paragraph element to obtain the first text content of each web page; if the tag name of the first node is not paragraph data, determining the central area according to the position data of the central point, merging the text content data of the nodes in the central area, and obtaining the second text content of each web page; Calculate the similarity between the first content and the second content in each web page respectively; Calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page; wherein the first parameter is the parameter of the external link pointing back to itself; According to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, in combination with a preset weighting algorithm, the score of each web page is obtained, and according to the scores of all web pages, each web page is classified into web page categories; wherein the web page categories include list pages and text pages; The method obtains the score of each web page according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page in combination with a preset weighting algorithm, and distinguishes the web page categories according to the scores of all web pages, specifically: weighted calculation is performed on the first PR value, the second PR value, the first parameter and the similarity of each web page, and the fixed weight value of each dimension is obtained according to the grid search method, so as to weight each dimension and obtain the score of each page; the scores of all pages are sorted from high to low, and the first N% of the web pages are determined as list pages, and the rest are determined as content pages; wherein N is a positive number.
2. The web page category determination method based on node extraction according to claim 1 is characterized in that: The central area is determined according to the position data of the central point, and the text contents of the nodes in the central area are merged, specifically: A rectangular central area is determined at the position of the central point; wherein the rectangular central area is delineated according to the mathematical golden ratio, r%=0.382, and the formula is: In the formula, centerX and centerY are the position data of the center point, width is the width data of the page, height is the height data of the page, and X i and Y i They are the width and height data of the center area respectively.
3. The web page category determination method based on node extraction according to claim 1 is characterized in that: The similarity between the first content and the second content in each web page is calculated respectively, specifically: Vectorizing the first content and the second content by using TFIDF; The similarity between the vectorized first content and the second content is calculated to obtain a first similarity, and the calculation formula is: Where, Sim(pi) is the first similarity, is the vectorization of the first content, C T(pi) Vectorization of the second content; The first similarity is processed to obtain a second similarity, and the calculation formula is: Yes′(p i )=1-Yes(p i ) In the formula, Sim′(p i ) is the second similarity, wherein the second similarity is the similarity between the first content and the second content.
4. The web page category determination method based on node extraction according to claim 1 is characterized in that: The method of calculating the PageRank value of each first content and the PageRank value of each second content respectively to obtain the first PR value and the second PR value corresponding to each web page is specifically as follows: Calculate the PageRank value of the first content to obtain a third PR value, and calculate the PageRank value of the second content to obtain a fourth PR value; The third PR value and the fourth PR value are normalized respectively to obtain a first PR value and a second PR value, wherein the normalization formula is: In the formula, PR′ o (p i ) is the first PR value, PR′(p i ) is the second PR value, PR o (p i ) is the third PR value, PR(p i ) is the fourth PR value, max(PR) and min(PR) are the maximum and minimum values of the corresponding second PR values of all web pages; max(PR o )、min(PR o ) are the maximum and minimum values of the corresponding first PR values of all web pages.
5. A web page category determination device based on node extraction, characterized in that: include: Information extraction module, similarity module, PR value calculation module and differentiation module; The information extraction module is used to extract multiple web page information, and obtain the first content and the second content of each web page according to the multiple web page information; wherein the first content and the second content are obtained at different time nodes; the extraction of multiple web page information, and obtaining the first content and the second content of each web page according to the multiple web page information, is specifically: according to the multiple web page information, extracting the width data, height data and multiple node data of each web page; wherein each node data includes the position data, label name data and text content data of a node; according to the width data and height data obtained in each web page, calculating the position data of the center point; according to the position data of the node obtained in each web page, calculating the distance data from the node to the center point; determining the first node according to the distance data in each web page, and obtaining the text content of each web page through the first node, as the first content of each web page; after a first preset time interval, re-extract the width data, height data and node data of the plurality of web pages to obtain the second content of each web page; wherein the text content includes: the first text content and the second text content; the first node is determined according to the distance data in each web page, and the text content of each web page is obtained through the first node, specifically: the node with the smallest distance data is selected as the first node in each web page; if the tag name data of the first node is a paragraph element, the text content data of all nodes with the tag name of the paragraph element are merged to obtain the first text content of each web page; if the tag name of the first node is not paragraph data, the central area is determined according to the position data of the central point, and the text content data of the nodes in the central area are merged to obtain the second text content of each web page; The similarity module is used to calculate the similarity between the first content and the second content in each web page respectively; The PR value calculation module is used to calculate the PageRank value of each first content and the PageRank value of each second content respectively, obtain the first PR value and the second PR value corresponding to each web page, and calculate the first parameter of the external link node in each web page; The distinguishing module is used to obtain the score of each web page according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page, combined with a preset weighting algorithm, and distinguish the web page category of each web page; wherein the web page category includes list page and text page; The method obtains the score of each web page according to the first PR value, the second PR value, the first parameter and the similarity corresponding to each web page in combination with a preset weighting algorithm, and distinguishes the web page categories according to the scores of all web pages, specifically: weighted calculation is performed on the first PR value, the second PR value, the first parameter and the similarity of each web page, and the fixed weight value of each dimension is obtained according to the grid search method, so as to weight each dimension and obtain the score of each page; the scores of all pages are sorted from high to low, and the first N% of the web pages are determined as list pages, and the rest are determined as content pages; wherein N is a positive number.
6. A computer terminal device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a method for determining a web page category based on node extraction as claimed in any one of claims 1 to 4 is implemented.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the web page category determination method based on node extraction as claimed in any one of claims 1 to 4.
