Web main content list identification method and apparatus

Abstract

The application relates to a webpage main content list identification method and device. The method comprises the following steps: obtaining a webpage to be identified; extracting webpage tags in the webpage to be identified; constructing a webpage tag combination comprising a preset number of webpage tags, finding a minimum parent node corresponding to the webpage tag combination, and constructing a candidate node list according to the minimum parent node; and obtaining a webpage main content list corresponding to the webpage to be identified according to the candidate node list. When the webpage main content is identified, the minimum parent nodes of multiple webpage tags in the webpage are mainly used to construct the candidate node list, then the webpage main content list is identified based on the candidate node list, and the method does not depend on a fixed frame, can effectively and automatically intelligently identify different webpages, the identification result is not affected by the change of a page structure, and the identification accuracy is ensured.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a method and apparatus for recognizing the main content list of a webpage. Background Technology

[0002] With the development of computer and web technologies, HTML (Hypertext Markup Language) technology emerged. HTML is specifically a markup language. It includes a series of tags. These tags unify the format of documents on the web, connecting scattered online resources into a logical whole. HTML text is descriptive text composed of HTML commands, which can specify text, graphics, animations, sounds, tables, links, etc. In an HTML document, the `<link>` tag is used to define hyperlinks, that is, to link from one page to another; in other words, `<link>` tags can be used to navigate between web pages.

[0003] Currently, specific algorithms can automatically identify the main content list information on an HTML page. However, current main content list identification methods can only capture main content list information within fixed frames. When the webpage structure changes, the accuracy of main content list identification cannot be guaranteed. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for identifying the main content list of web pages that can effectively guarantee the accuracy of recognition, in order to address the above-mentioned technical problems.

[0005] A method for identifying the main content list of a webpage, the method comprising:

[0006] Obtain the webpage to be identified;

[0007] Extract web page tags from the web page to be identified; these web page tags are used to define anchors or hyperlinks.

[0008] Construct a webpage tag combination that includes a preset number of webpage tags, find the smallest parent node corresponding to the webpage tag combination, and construct a candidate node list based on the smallest parent node;

[0009] The list of main content of the webpage corresponding to the webpage to be identified is obtained from the list of candidate nodes.

[0010] A webpage main content list recognition device, the device comprising:

[0011] The webpage acquisition module is used to acquire the webpage to be identified.

[0012] The tag extraction module is used to extract web page tags from the web page to be identified, and the web page tags are used to define anchors or hyperlinks.

[0013] The list building module is used to build a combination of web page tags including a preset number of web page tags, find the smallest parent node corresponding to the combination of web page tags, and build a list of candidate nodes based on the smallest parent node;

[0014] The main content acquisition module is used to obtain the main content list of the webpage corresponding to the webpage to be identified based on the candidate node list.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0016] Obtain the webpage to be identified;

[0017] Extract web page tags from the web page to be identified; these web page tags are used to define anchors or hyperlinks.

[0018] Construct a webpage tag combination that includes a preset number of webpage tags, find the smallest parent node corresponding to the webpage tag combination, and construct a candidate node list based on the smallest parent node;

[0019] The list of main content of the webpage corresponding to the webpage to be identified is obtained from the list of candidate nodes.

[0020] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0021] Obtain the webpage to be identified;

[0022] Extract web page tags from the web page to be identified; these web page tags are used to define anchors or hyperlinks.

[0023] Construct a webpage tag combination that includes a preset number of webpage tags, find the smallest parent node corresponding to the webpage tag combination, and construct a candidate node list based on the smallest parent node;

[0024] The list of main content of the webpage corresponding to the webpage to be identified is obtained from the list of candidate nodes.

[0025] The aforementioned method, apparatus, computer equipment, and storage medium for identifying the main content list of a webpage involve: acquiring a webpage to be identified; extracting webpage tags from the webpage; constructing a combination of webpage tags including a preset number of tags; finding the smallest parent node corresponding to the combination of tags; constructing a candidate node list based on the smallest parent node; and obtaining the main content list of the webpage to be identified based on the candidate node list. In identifying the main content of a webpage, this application primarily constructs a candidate node list by identifying the smallest parent node of multiple webpage tags within the webpage, and then identifies the main content list based on this candidate node list. It does not rely on a fixed frame, effectively enabling automated intelligent identification for different webpages, and the identification results are not affected by changes in the page structure, ensuring high accuracy. Attached Figure Description

[0026] Figure 1 This is an application environment diagram of a webpage main content list recognition method in one embodiment;

[0027] Figure 2 This is a flowchart illustrating a method for identifying the main content list of a webpage in one embodiment;

[0028] Figure 3 This is a flowchart illustrating the steps for removing abnormal tags from webpage tags in one embodiment.

[0029] Figure 4 This is a flowchart illustrating the steps of constructing a list of candidate nodes in one embodiment;

[0030] Figure 5 This is a flowchart illustrating the steps of obtaining the main content list of the webpage corresponding to the webpage to be identified in one embodiment;

[0031] Figure 6 This is a structural block diagram of a webpage main content list recognition device in one embodiment;

[0032] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] The webpage main content list identification method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can provide server 104 with data related to the webpage to be identified, and then the server identifies the main content list within the webpage. The link address of the webpage to be identified is provided by terminal 102. The server obtains the webpage to be identified; extracts webpage tags from the webpage to be identified, the webpage tags being used to define anchors; finds the smallest parent node corresponding to a webpage tag combination including a preset number of webpage tags, constructs a candidate node list based on the smallest parent node; and obtains the main content list of the webpage corresponding to the webpage to be identified based on the candidate node list. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. In another embodiment, the webpage main content list identification method of this application can also be applied to a terminal and implemented through the terminal.

[0035] In one embodiment, such as Figure 2 As shown, a method for recognizing the main content list of a webpage is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0036] Step 201: Obtain the webpage to be identified.

[0037] The webpage to be identified refers to the object of identification by the webpage main content list identification method of this application. Specifically, the webpage to be identified is a web page, i.e., an HTML page. The webpage to be identified contains several content blocks, which record the specific content of the webpage. The main content corresponding to the webpage to be identified can be identified by the webpage main content list identification method of this application.

[0038] Specifically, this application can identify the main content list of a webpage using JavaScript-based webpage manipulation methods. Specifically, this application uses a DOM tree approach for webpage manipulation, where DOM stands for Document Object Model, the interface for JavaScript to manipulate webpages. The DOM tree is a tree composed of HTML page content structure nodes arranged from top to bottom. First, when identifying the main content of a webpage, server 104 needs to obtain the webpage to be identified. This webpage is submitted by terminal 102, and terminal 102 can submit a webpage link. Then, server 104 directly obtains the rendered webpage content based on the webpage link (ensuring that single-page server-rendered pages can also obtain webpage data). Puppeteer or Chromium can typically be used for crawling to ensure the integrity of the page content. When server 104 creates the corresponding webpage to be identified based on the webpage link and loads it into a web browser, the corresponding DOM tree is also generated. This application identifies the main content list of the webpage based on operations on the DOM tree and its nodes.

[0039] Step 203: Extract web page tags from the web page to be identified. Web page tags are used to define anchors or hyperlinks.

[0040] In this context, the webpage tag, specifically an HTML page tag, is used to define hyperlinks (used to link from one page to another) or anchors (used to jump from the current position on the page to a specified anchor point). Its most important attribute is the `href` attribute, which indicates the target of the link. In one embodiment, the webpage tag in this application is specifically an anchor tag, also known as an A tag, which is supported by all browsers. Anchor tags are used for events such as anchor point jumps, link jumps, button clicks, and link downloads. The anchor tag, also known as an A tag, has the format: content This application primarily determines the smallest parent node based on the relationships between webpage tags in order to identify the main content of the webpage. Therefore, the webpage tags in the webpage to be identified must be extracted first during the identification process.

[0041] Specifically, when performing webpage tag recognition, JavaScript can be used to extract webpage tags from the webpage to be recognized. For example, in a specific embodiment, webpage tags can be extracted from the rendered HTML page using regular expressions. Simultaneously, to improve the accuracy of subsequent content recognition, anomaly detection can be performed during tag extraction to filter out abnormal tags from the extracted webpage tags.

[0042] Step 205: Construct a webpage tag combination that includes a preset number of webpage tags, find the smallest parent node corresponding to the webpage tag combination, and construct a candidate node list based on the smallest parent node.

[0043] In the HTML DOM tree, according to the W3C (World Wide Web Consortium) HTML DOM standard, all content in an HTML document is a node: the entire document is a document node, each HTML element is an element node, the text within an HTML element is a text node, each HTML attribute is an attribute node, and comments are comment nodes. The HTML DOM views the HTML document as a tree structure. This structure is called a node tree, therefore, a webpage tag can be considered a node. Nodes in the node tree have hierarchical relationships. These relationships are typically described using terms such as parent, child, and sibling. A parent node has child nodes. Child nodes at the same level are called siblings. Therefore, a webpage tag can be considered a child node, and the node above it in the corresponding hierarchical relationship can be considered a parent node. In the node tree, the top node is called the root. Every node has a parent node, except for the root (which has no parent node). A node can have any number of child nodes. Siblings are nodes that share the same parent node. The preset number of minimum parent nodes corresponding to webpage tags specifically refers to finding the minimum parent node for multiple different webpage tags. This could be the parent node of these tags, or even the parent node of the parent node of these webpage tags. A list of candidate nodes is then constructed based on these found parent nodes. In this application, a list refers to a group of nodes with the same node type, internal structure, and the same parent node. The list discussed here generally refers to the parent node of nodes of the same type and internal structure. The preset number can be 2, 3, 4, 5, etc., but through experiments, when the preset number is 3, the content recognition accuracy of the main webpage content list is the highest, and the scheme has the best performance.

[0044] Specifically, this application primarily identifies the main content of a webpage by filtering the smallest parent node of webpage tags, and simultaneously constructs a corresponding list of main webpage content. During the identification process, webpage tags are first obtained, and then the parent nodes corresponding to those tags are found. For example, in a specific embodiment, the preset number is 3, and the identified webpage tags include tags 1, 2, 3, 4, 5, and 6. In this case, if the preset number is 3, it is necessary to first construct ten combinations of webpage tags: {1, 2, 3}, {1, 2, 4}, {1, 2, 5}, {1, 2, 6}, {1, 3, 4}, {1, 3, 5}, {1, 3, 6}, {1, 4, 5}, {1, 4, 6}, and {1, 5, 6}, and then find the smallest parent node corresponding to these 10 combinations. Then, based on these 10 found parent nodes, a candidate node list is constructed. This process also requires deduplication of parent nodes in the candidate node list.

[0045] Step 207: Obtain the list of main content of the webpage corresponding to the webpage to be identified based on the list of candidate nodes.

[0046] Specifically, after obtaining the candidate node list, since the number of nodes in the candidate list may be greater than one, while the main content of the webpage should only correspond to the main content, this application needs to further filter the candidate node list. In one embodiment, this filtering can be based on the height of different nodes in the candidate node list. This results in a list of the main content of the webpage corresponding to the webpage to be identified. The obtained list of the main content specifically includes the webpage's main content and consists of a list of pages with the same page structure.

[0047] In one embodiment, such as Figure 3 As shown, step 207 includes:

[0048] Step 302: When the nodes in the candidate node list are not unique, identify the abnormal nodes in the candidate node list.

[0049] Step 304: Remove abnormal nodes from the candidate node list.

[0050] Step 306: Based on the list of candidate nodes after removing abnormal nodes, obtain the list of main content of the webpage corresponding to the webpage to be identified.

[0051] Abnormal nodes include those with abnormal positions and those with abnormal heights. Specifically, when identifying the main content, it is also necessary to exclude the smallest parent node of non-main content parts to filter out more valuable main content list nodes. Therefore, when the nodes in the candidate node list are not unique, abnormal nodes in the candidate node list can be identified first, and then these abnormal nodes can be removed before performing the corresponding main content list identification. In this embodiment, by removing abnormal nodes from the candidate node list, the identification accuracy of the main content list can be effectively improved.

[0052] In one embodiment, step 308 includes: obtaining the list length of the candidate node list after removing abnormal nodes; and identifying the main content list of the webpage corresponding to the webpage to be identified based on the list length.

[0053] Specifically, the length of the candidate node list after removing abnormal nodes refers to the number of nodes in the candidate node list. Generally, a webpage only includes one theme, meaning the main content list of the webpage should also contain only one node. Therefore, the final method for identifying the main content list of the webpage can be determined by the length of the candidate node list. In this embodiment, by identifying the main content list of the webpage to be identified based on the list length, the main content list can be selected more effectively for accurate identification of the main content of the webpage.

[0054] In one embodiment, identifying the main content list of the webpage corresponding to the webpage to be identified based on the list length includes: when the list length indicates that the nodes in the candidate node list are unique, the nodes in the candidate node list are taken as the main content list of the webpage corresponding to the webpage to be identified; when the list length indicates that the nodes in the candidate node list are not unique, the height of each node in the candidate node list after removing abnormal nodes is obtained, and the node with the largest height is taken as the main content list of the webpage corresponding to the webpage to be identified.

[0055] Specifically, the list of web pages is a list of pages composed of the same page structure. The list involved in this application generally refers to the parent node of nodes of the same type and internal structure. The elements contained in the main content list of the web page are the aforementioned minimum parent nodes, which can be used as the identification result of the main content list identification method. Therefore, when the list length indicates that the nodes in the candidate node list are unique (i.e., the list length is 1), the candidate node list after removing abnormal nodes contains only one node, i.e., only one minimum parent node. Therefore, the nodes in the candidate node list after removing abnormal nodes can be used as the main content list of the web page corresponding to the web page to be identified. When the list length indicates that the nodes in the candidate node list are not unique (i.e., the list length is greater than 1), the height of the nodes can be compared to determine which node corresponds to the main content list of the web page. Specifically, the height of each node in the candidate node list after removing abnormal nodes can be obtained, and then the node with the largest height can be selected by comparing the heights of different nodes. The node with the largest height is then used as the main content list of the web page corresponding to the web page to be identified. The reason for selecting the node with the largest height as the main content node is that the height of a node in the DOM tree specifically refers to the content definition height, i.e., the height of the content of the element corresponding to the node. Generally, the height of the main content of a webpage is within a relatively fixed range, and comparing node heights can demonstrate the importance of the content within different nodes. Therefore, the main content of a webpage can be identified based on node height. Furthermore, as the carrier of webpage content, the main content presented to the user in an HTML webpage typically occupies the largest portion of the page, while other rendering and advertising content occupy a smaller portion. Since node height, i.e., content definition height, can be considered the height of the content's space, the node with the largest height can be used as the main content list for the webpage to be identified. Specifically, when the list length is 0, meaning the candidate node list after removing abnormal nodes contains no nodes, it can be determined that the webpage to be identified does not have a corresponding main content list. In this embodiment, identifying the main content list of the webpage to be identified by using the length of the candidate node list and the candidate node list itself can effectively improve the accuracy of the identification.

[0056] In one embodiment, the abnormal nodes include location abnormal nodes, and step 302 includes: when the nodes in the candidate node list are not unique, obtaining the vertical center line of the document corresponding to the webpage to be identified; when the vertical center line does not pass through the content block of the node in the candidate node list, the node whose vertical center line does not pass through the content block is determined as a location abnormal node.

[0057] The vertical center line of the document corresponding to the webpage to be identified refers to the vertical center line of the HTML document page of the webpage to be identified, with the top and bottom edges of the HTML document page at their respective ends. Content blocks containing the main content of the webpage will undoubtedly cross the vertical center line of the HTML document page, making it easier for readers to grasp the main content. Therefore, whether a content block crosses the vertical center line of the document corresponding to the webpage to be identified can be used as a criterion. When a content block crosses the vertical center line of the HTML document page, it is not necessarily the main content of the webpage; however, when a content block does not cross the vertical center line of the HTML document page, it is not necessarily the main content of the webpage. Therefore, the identification of the main content of the webpage can be optimized by excluding nodes corresponding to content blocks that do not cross the vertical center line of the HTML document page from the list of candidate nodes.

[0058] Specifically, when identifying the main content, it's necessary to exclude the smallest parent node of non-main content sections and filter out more valuable main content list nodes. During this process, positional anomalies can be identified using the vertical center line of the HTML document page of the webpage to be identified. Generally, for webpage layouts, the main content is placed in the center of the document to provide the optimal reading experience for readers, while the left and right sides are occupied by non-main content such as backlinks or advertisements. Therefore, positional anomalies can be determined based on whether the content blocks of nodes in the candidate node list cross the vertical center line, and nodes not in the center of the page are considered abnormal and removed. In this embodiment, identifying abnormal nodes in the candidate node list by their position can effectively improve the accuracy of main content list identification.

[0059] In one embodiment, abnormal nodes include height abnormal nodes. When the nodes in the candidate node list are not unique, identifying abnormal nodes in the candidate node list includes: when the nodes in the candidate node list are not unique, obtaining the height of the nodes in the candidate node list; when the height of the nodes in the candidate node list is lower than a preset height threshold, determining the nodes whose height is lower than the preset height threshold as height abnormal nodes.

[0060] Specifically, when identifying the main content, it's necessary to exclude the smallest parent node of non-main content sections and filter out more valuable main content list nodes. In this process, height anomalies can be identified by the height of nodes within the candidate node list. For nodes whose height is below a preset height threshold, even if the corresponding content block is located in the center of the webpage, its content may not necessarily be the main content. At the top or bottom of the webpage document, it's generally a non-main content section such as backlinks or advertisements. Since these sections typically have lower node heights, they can also be removed as abnormal nodes, thus obtaining the final main content list of the webpage to be identified. The specific value of the preset height threshold can be optimized through experiments. For example, in one embodiment, the preset height threshold can be set to 200px; when the height of a node in the candidate node list is below 200px, it can be considered an abnormal node. In this embodiment, identifying abnormal nodes within the candidate node list by their height can effectively improve the accuracy of main content list identification.

[0061] The aforementioned method for recognizing the main content list of a webpage involves: acquiring the webpage to be recognized; extracting webpage tags from the webpage; constructing a combination of webpage tags including a preset number of tags; finding the smallest parent node corresponding to the combination of tags; constructing a candidate node list based on the smallest parent node; and obtaining the main content list of the webpage to be recognized based on the candidate node list. This application, when recognizing the main content of a webpage, primarily constructs a candidate node list by identifying the smallest parent node of multiple webpage tags in the webpage, and then recognizes the main content list based on this candidate node list. It does not rely on a fixed frame, effectively enabling automated intelligent recognition for different webpages, and the recognition results are not affected by changes in the page structure, ensuring high accuracy.

[0062] In one embodiment, before step 205, the method further includes: identifying anomalous tags among multiple webpage tags; and removing anomalous tags from the multiple webpage tags. Step 205 includes: constructing a webpage tag combination that includes a preset number of webpage tags after removing hidden tags.

[0063] Abnormal tags refer to tags that cannot be considered part of the normal content of a webpage. As shown above, the `<link>` tag is an HTML page tag used to define hyperlinks (used to link from one page to another) or anchors (used to jump from the current position on the page to a specified anchor point). Its most important attribute is the `href` attribute, which indicates the target of the link. When a webpage tag is abnormal, such as a hidden tag or a tag that cannot be displayed correctly, it cannot display the corresponding webpage content. In this case, it should not be used as the basis for identifying the main content of the webpage. Therefore, abnormal tags can be identified and removed before constructing the webpage tag combination, thereby improving the accuracy of subsequent extraction of the main content of the webpage.

[0064] In one embodiment, such as Figure 4 As shown, abnormal tags include tags indicating display issues. Identifying abnormal tags among multiple webpage tags includes:

[0065] Step 401: Obtain the display effect data of web page tags. The display effect data includes one or more of the following: content length information, display height information, and display width information.

[0066] Step 403: Identify abnormal tags in the webpage tags based on the display effect data.

[0067] Specifically, the content length information of a webpage tag refers to the character length of the content it contains. If the character length is lower than a preset content length threshold, the webpage tag is considered abnormal and cannot be used to identify the main content list of the webpage. Furthermore, if the display height or width information of a webpage tag is also abnormal, it can also be considered an abnormal webpage tag. For example, a webpage tag with a width or height of zero cannot be displayed on a normal webpage, and readers cannot obtain any information from it; therefore, it is undoubtedly an abnormal webpage tag.

[0068] Specifically, after extracting webpage tags from the webpage to be identified, and before finding the smallest parent node corresponding to each tag, abnormal tags can be identified to improve the accuracy of the main content list recognition. Besides determining whether a tag is hidden, identification can also be based on the attributes of the webpage tags themselves. Anomaly identification is performed based on the length of the webpage tags themselves, and the length or width of the tags displayed on the webpage. In a specific embodiment, webpage tags with content less than a preset character threshold, or tags with zero width or height when displayed, are removed to obtain a valid list of webpage tags. Regarding abnormal content length, after multiple experiments and observations, it was determined that when the content of a webpage tag is less than 4 characters, the corresponding content is generally an advertisement or external link. Therefore, the preset content length threshold can be set to 4 characters, at which point the accuracy of the main content list recognition is highest. In this embodiment, by identifying and removing abnormal tags after extracting webpage tags before performing main content list recognition, the accuracy of the main content list recognition can be effectively improved.

[0069] In one embodiment, the abnormal tags include hidden tags. Removing abnormal tags from multiple web page tags includes: when there are web page tags that belong to hidden tags among the multiple web page tags, removing the hidden tags from the multiple web page tags, and the hidden tags are not displayed in the web page to be identified; step 205 includes: constructing a web page tag combination including a preset number of web page tags after removing hidden tags.

[0070] Hidden tags are tags that are not displayed in the content of a webpage. Many tags in HTML are not displayed, such as: <meta content> , <script>、<STYLE>等，在这些标签中堆砌一些希望获得排名的关键词，可以提高关键字密度，而且不影响阅读。隐藏标签包括隐藏文字，隐藏页面以及隐藏链接等。其中隐藏文字是指在网页的HTML文件中放上含有关键词的文字，但这些字不能被用户所看到，只有搜索引擎才能进行读取，一般有超小字号的文字，与背景同样颜色的文字，放在评论标签当中的文字，放在表格input标签里面的文字，通过样式表把文字放在不可见的层上面等等。其目的也都是想提高网页的相关性。有的时候，有的人还在这些地方放上与网站内容无关的，但是很热门的关键词，希望网页能在这些热门关键词下得到好的排名和流量。隐藏页面是一种障眼法的优化，让普通用户和搜索引擎看到的页面是不同的版本，如果是搜索引擎，则返回一个高度优化，甚至优化到已经没法读的页面；如果是正常用户，则返回一个不考虑seo、给用户看的正常页面。隐藏链接和隐藏文字相似，但是区别是把关键词放在链接里面，而这个链接也是用户所看不到的。

[0071] 具体地，在提取待识别网页中的网页标签之后，查找网页标签对应的最小父节点之前，为了提高主内容列表的识别准确率，可以进行异常标签的识别，首先对于隐藏标签，因为这部分标签不是显示给网页用户看到的，因此不会参与网页主题，即与网页主内容列表识别无关，因此可以将其作为异常标签进行排除。本实施例中，通过在提取网页标签后，识别并剔除出其中包含的隐藏标签后再进行页主内容列表识别，可以有效提高网页主内容列表识别的准确率。

[0072] 其中一个实施例中，如图5所示，步骤205包括：

[0073] 步骤502，查找包括预设数目网页标签的网页标签组合对应的最小父节点。

[0074] 步骤504，对查找到的最小父节点进行去重处理，根据去重处理结果构建待选节点列表。

[0075] 其中，去重主要用于去除待选节点列表中重复出现的最小父节点，从而保证网页主内容列表识别的准确率。具体的，对于得到的，预设数目的网页标签对应的最小父节点。可能存在多组网页标签对应同一个网页标签的情况，因此构建的待选节点列表内会出现多个相同的父节点的情况。此时，为了避免后续处理过程出现错误，可以在查找到最小父节点后，直接识别查找到的所有最小父节点中重复的节点，而后进行去重处理，对于多个相同的父节点，仅保留一个，删去多余的重复节点。

[0076] 具体地，在查找到网页中所有的网页标签组合对应最小父节点后，为了防止出现最小父节点的重复，可以进行去重处理。在其中一个实施例中，可以通过HTML DOM的isSameNode()方法，对所得到的最小父节点进行两两直接的判断，识别其中重复的最小父节点。isSameNode()方法可在某个节点与给定的节点是同一个节点时返回true，否则返回false。可以有效地识别出所得最小父节点中的重复节点，从而进行有效去重。本实施例中，通过对最小父节点进行去重处理，可以有效保证待选节点列表构建的准确性。

[0077] 本申请还提供一种应用场景，该应用场景应用上述的网页主内容列表识别方法。具体地，该网页主内容列表识别方法在该应用场景的应用如下：

[0078] 当进行新闻网站的网页内容的统计时，终端方需要确定指定内容的新闻都已发布到网页页面上，因此需要对指定网页内的网页主内容列表进行识别，此时终端可以提交这些新闻的网页链接至服务器，有服务器来对这些新闻的网页页面内网页主内容列表进行识别。在进行识别时，服务器依次对各个网页内的网页主内容列表进行识别。在针对单个网页页面进行识别时，服务器首先基于获取的网页链接，使用puppeteer获取渲染之后的待识别网页的网页内容，而后提取出该网页页面内所有的网页标签，并且剔除其中不正常的网页标签，这里不正常的网页标签包括隐藏标签、网页标签内容长度少于4个字符、标签显示的时候宽或者是高为零的网页标签等等，剔除之后得到有效的网页标签列表。在有效的网页标签列表中，以每三个网页标签为一个序列，找出三个网页标签公共的最小父节点，而后将所获得的所有父节点去重之后得到待选节点列表，如果得到的待选节点列表内节点不唯一，则在进行待选节点列表内节点的筛选，包括：(1)获取节点的高度，高度小于固定阈值200px的，将其剔除；(2)获取待识别网页对应HTML文档整个文档的宽度，基于宽度找到HTML文档的垂直中心线，保证垂直中心线穿过待选节点列表内节点的内容块，没有穿过的，将其剔除出待选节点列表。而后最终得到的列表来进行网页主内容识别，如果列表长度为0，则表示待识别网页没有对应的主内容列表；如果列表长度为1，则里面的那个节点就是待识别网页的主内容列表；如果列表长度大于1，则再遍历待选节点列表，来获取其中每个节点的高度，取高度最大的为待识别网页的页面主内容列表。

[0079] 应该理解的是，虽然上述流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，上述流程图中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。

[0080] 在一个实施例中，如图6所示，提供了一种网页主内容列表识别装置600，该装置可以采用软件模块或硬件模块，或者是二者的结合成为计算机设备的一部分，该装置具体包括：网页获取模块601、标签提取模块603、列表构建模块605和主内容获取模块607，其中：

[0081] 网页获取模块601，用于获取待识别网页。

[0082] 标签提取模块603，用于提取待识别网页中的网页标签，网页标签用于定义锚或超链接。

[0083] 列表构建模块605，用于构建包括预设数目网页标签的网页标签组合，查找网页标签组合对应的最小父节点，根据最小父节点构建待选节点列表。

[0084] 主内容获取模块607，用于根据待选节点列表获取待识别网页对应的网页主内容列表。

[0085] 在其中一个实施例中，还包括异常标签识别模块，用于：识别多个网页标签中的异常标签；剔除多个网页标签中的异常标签；列表构建模块605具体用于：构建包括预设数目的剔除隐藏标签后的网页标签的网页标签组合。

[0086] 在其中一个实施例中，异常标签识别模块还用于：获取网页标签的显示效果数据，显示效果数据包括内容长度信息、显示高度信息以及显示宽度信息中的一个或多个；根据显示效果数据识别网页标签中的异常标签。

[0087] 在其中一个实施例中，异常标签识别模块还用于：当多个网页标签中存在属于隐藏标签的网页标签时，剔除多个网页标签中的隐藏标签，隐藏标签未显示于待识别网页中；列表构建模块605具体用于：构建包括预设数目的剔除隐藏标签后的网页标签的网页标签组合。

[0088] 在其中一个实施例中，列表构建模块605还用于：查找标签组合对应的最小父节点；对查找到的最小父节点进行去重处理，根据去重处理结果构建待选节点列表。

[0089] 在其中一个实施例中，主内容获取模块607具体用于：当待选节点列表内节点不唯一时，识别待选节点列表中的异常节点；剔除待选节点列表内的异常节点；根据剔除异常节点后的待选节点列表，获取待识别网页对应的网页主内容列表。

[0090] 在其中一个实施例中，主内容获取模块607还用于：获取剔除异常节点后的待选节点列表的列表长度；根据列表长度识别待识别网页对应的网页主内容列表。

[0091] 在其中一个实施例中，主内容获取模块607还用于：当列表长度表征待选节点列表内节点唯一时，将待选节点列表内的节点作为待识别网页对应的网页主内容列表；当列表长度表征待选节点列表内节点不唯一时，获取剔除异常节点后的待选节点列表中每个节点的高度，将高度最大的节点作为待识别网页对应的网页主内容列表。

[0092] 在其中一个实施例中，异常节点包括位置异常节点，主内容获取模块607还用于：当待选节点列表内节点不唯一时，获取待识别网页对应文档的垂直中心线；当垂直中心线未穿过待选节点列表内节点的内容块时，将垂直中心线未穿过内容块的节点判定为位置异常节点。

[0093] 在其中一个实施例中，异常节点包括高度异常节点，主内容获取模块607还用于：当待选节点列表内节点不唯一时，获取待选节点列表内节点的高度；当待选节点列表内节点的高度低于预设高度阈值时，将节点的高度低于预设高度阈值的节点判定为高度异常节点。

[0094] 关于网页主内容列表识别装置的具体实施例可以参见上文中对于网页主内容列表识别方法的实施例，在此不再赘述。上述网页主内容列表识别装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。

[0095] 在一个实施例中，提供了一种计算机设备700，该计算机设备700可以是服务器，其内部结构图可以如图7所示。该计算机设备700包括通过系统总线701连接的处理器702、存储器和网络接口704。其中，该计算机设备700的处理器702用于提供计算和控制能力。该计算机设备700的存储器包括非易失性存储介质705、内存储器703。该非易失性存储介质705存储有操作系统706、计算机程序707和数据库708。该内存储器703为非易失性存储介质705中的操作系统706和计算机程序707的运行提供环境。该计算机设备的数据库708用于存储输入的待分类图数据以及数据分类方法执行过程中的运行数据。该计算机设备的网络接口704用于与外部的终端通过网络连接通信。该计算机程序707被处理器702执行时以实现一种数据分类方法。

[0096] 本领域技术人员可以理解，图7中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。

[0097] 在一个实施例中，还提供了一种计算机设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现上述各方法实施例中的步骤。

[0098] 在一个实施例中，提供了一种计算机可读存储介质，存储有计算机程序，该计算机程序被处理器执行时实现上述各方法实施例中的步骤。

[0099] 在一个实施例中，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述各方法实施例中的步骤。

[0100] 本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-OnlyMemory，ROM)、磁带、软盘、闪存或光存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或外部高速缓冲存储器。作为说明而非局限，RAM可以是多种形式，比如静态随机存取存储器(Static Random Access Memory，SRAM)或动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。

[0101] 以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。

[0102] 以上实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。< / script>

Claims

1. A method for recognizing the main content list of a webpage, characterized in that, The method includes: Obtain the webpage to be identified; Extract multiple web page tags from the web page to be identified, where the web page tags are used to define anchors or hyperlinks; The display effect data of the webpage tags is obtained, including one or more of content length information, display height information, and display width information; based on the display effect data, abnormal display effect tags are identified in the webpage tags, including webpage tags whose content length is less than a preset character count threshold, whose display height is zero, or whose display width is zero; when there are webpage tags that are hidden tags among the multiple webpage tags, the hidden tags are removed from the multiple webpage tags, and the hidden tags are not displayed in the webpage to be identified; For the web page tags after removing the abnormal display effect tags and the hidden tags, a full combination enumeration is performed according to a preset number to generate multiple web page tag combinations. The minimum parent node corresponding to the web page tag combination is found. The minimum parent nodes found are compared pairwise by the node identity determination. Duplicate nodes are identified among the found minimum parent nodes and deduplication is performed. A candidate node list is constructed based on the deduplication result. The minimum parent node corresponding to the web page tag combination is the minimum parent node common to the web page tags in the web page tag combination. When the nodes in the candidate node list are not unique, identify abnormal nodes in the candidate node list; remove abnormal nodes from the candidate node list; and obtain the main content list of the webpage corresponding to the webpage to be identified based on the candidate node list after removing abnormal nodes. The abnormal nodes include nodes with abnormal positions and nodes with abnormal heights; The method for identifying abnormally positioned nodes includes: when the nodes in the candidate node list are not unique, obtaining the vertical center line of the document corresponding to the webpage to be identified; when the vertical center line does not pass through the content block of the node in the candidate node list, determining the node whose vertical center line does not pass through the content block as an abnormally positioned node. The method for identifying height-abnormal nodes includes: when the nodes in the candidate node list are not unique, obtaining the height of the nodes in the candidate node list; when the height of the nodes in the candidate node list is lower than a preset height threshold, determining the nodes with heights lower than the preset height threshold as height-abnormal nodes.

2. The method according to claim 1, characterized in that, The hidden tags include hidden text, hidden pages, and hidden links.

3. The method according to claim 1, characterized in that, The step of obtaining the main content list of the webpage corresponding to the webpage to be identified based on the candidate node list after removing abnormal nodes includes: Get the length of the candidate node list after removing abnormal nodes; The main content list of the webpage corresponding to the webpage to be identified is determined based on the length of the list.

4. The method according to claim 3, characterized in that, The step of identifying the main content list of the webpage corresponding to the webpage to be identified based on the list length includes: When the length of the list indicates that the nodes in the candidate node list are unique, the nodes in the candidate node list are taken as the main content list of the webpage corresponding to the webpage to be identified. When the length of the list indicates that the nodes in the candidate node list are not unique, the height of each node in the candidate node list after removing abnormal nodes is obtained, and the node with the largest height is taken as the main content list of the webpage corresponding to the webpage to be identified.

5. A webpage main content list recognition device, characterized in that, The device includes: The webpage acquisition module is used to acquire the webpage to be identified. The tag extraction module is used to extract web page tags from the web page to be identified, and the web page tags are used to define anchors or hyperlinks. An abnormal tag identification module is used to acquire display effect data of the webpage tags, the display effect data including one or more of content length information, display height information, and display width information; identify abnormal display tags in the webpage tags based on the display effect data, the abnormal display tags including webpage tags whose content length is less than a preset character count threshold, whose display height is zero, or whose display width is zero; when there are webpage tags that are hidden tags among the multiple webpage tags, remove the hidden tags from the multiple webpage tags, the hidden tags not displayed in the webpage to be identified; The list construction module is used to perform a full combination enumeration on the web page tags after removing the abnormal display effect tags and the hidden tags, generate multiple web page tag combinations, find the minimum parent node corresponding to the web page tag combination, compare the found minimum parent nodes pairwise through node identity determination, identify duplicate nodes among the found minimum parent nodes, and perform deduplication processing, construct a candidate node list based on the deduplication processing result, and the minimum parent node corresponding to the web page tag combination is the minimum parent node common to the web page tags in the web page tag combination; The main content acquisition module is used to identify abnormal nodes in the candidate node list when the nodes in the candidate node list are not unique; remove abnormal nodes from the candidate node list; and obtain the main content list of the webpage corresponding to the webpage to be identified based on the candidate node list after removing abnormal nodes. The abnormal nodes include nodes with abnormal positions and nodes with abnormal heights; The main content acquisition module is specifically used for: when the nodes in the candidate node list are not unique, acquiring the vertical center line of the document corresponding to the webpage to be identified; when the vertical center line does not pass through the content block of the node in the candidate node list, determining the node whose vertical center line does not pass through the content block as an abnormal position node. The main content acquisition module is also used to: when the nodes in the candidate node list are not unique, acquire the height of the nodes in the candidate node list; when the height of the nodes in the candidate node list is lower than a preset height threshold, determine the nodes whose height is lower than the preset height threshold as nodes with abnormal height.

6. The apparatus according to claim 5, characterized in that, The hidden tags include hidden text, hidden pages, and hidden links.

7. The apparatus according to claim 5, characterized in that, The main content acquisition module is specifically used to: acquire the length of the candidate node list after removing abnormal nodes; and identify the main content list of the webpage corresponding to the webpage to be identified based on the list length.

8. The apparatus according to claim 7, characterized in that, The main content acquisition module is specifically used for: when the list length indicates that the nodes in the candidate node list are unique, taking the nodes in the candidate node list as the main content list of the webpage corresponding to the webpage to be identified; when the list length indicates that the nodes in the candidate node list are not unique, obtaining the height of each node in the candidate node list after removing abnormal nodes, and taking the node with the largest height as the main content list of the webpage corresponding to the webpage to be identified.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

11. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions such that the computer device performs the steps of the method according to any one of claims 1 to 4.

Images