Webpage main body information extraction method and device and medium

Abstract

The invention discloses a webpage main body information extraction method and device and a medium, and relates to the technical field of Internet information processing. The method comprises the following steps: acquiring a document object model (DOM) tree of a webpage and visual rendering information of at least one HTML element through a headless browser; based on the visual rendering information, dividing the webpage into at least one visual block by using a visual separation algorithm, and calculating a visual feature score of the visual block; performing weighted fusion on the visual feature scores of the DOM nodes in the DOM tree and the structural feature scores of the DOM nodes to obtain comprehensive weight scores of the DOM nodes; and determining the DOM node with the comprehensive weight score greater than a preset weight threshold as a container node of the main body content, so as to perform information extraction on the container node and obtain main body information of the webpage. Therefore, webpage main body information extraction with high accuracy, good robustness and excellent efficiency can be realized.

Description

Technical Field

[0001] This application relates to the field of Internet information processing technology, and in particular to a method, device and medium for extracting main information from a webpage. Background Technology

[0002] With the explosive growth of internet information, automatically extracting the main content (such as news articles and product descriptions) from web pages has become a key technology for applications such as web crawlers, search engines, and data aggregation.

[0003] Existing mainstream extraction technologies are mainly divided into two categories: One approach is based on the Document Object Model (DOM) structure. This method analyzes the HTML DOM tree of a webpage and uses pre-defined rules or machine learning models (based on features such as tags, class names, and IDs) to locate the main content. Its drawbacks are significant: it is extremely dependent on a specific DOM structure. Once the website is redesigned, and HTML tags or style class names change, the pre-defined rules immediately become invalid, requiring manual re-adaptation, which is costly to maintain. Furthermore, a large amount of noisy content on the page, such as navigation bars, advertisements, and related recommendations, may resemble the main text in the DOM structure, leading to incorrect extraction.

[0004] Another approach is based on computer vision, which treats the entire webpage as an image and uses layout analysis algorithms to identify the main content areas. This method is unaffected by changes in the DOM structure and is relatively robust. However, it is computationally intensive, slow, and cannot meet the needs of large-scale web crawling, and it is difficult to perform accurate text-level extraction. Summary of the Invention

[0005] This application provides a method, device, and medium for extracting webpage main information, in order to solve the following technical problem: how to achieve high accuracy, robustness, and efficiency in extracting webpage main information.

[0006] In a first aspect, embodiments of this application provide a method for extracting main information of a webpage. The method includes: obtaining the Document Object Model (DOM) tree of the webpage and visual rendering information of at least one HTML element through a headless browser; dividing the webpage into at least one visual block based on the visual rendering information using a visual segmentation algorithm, and calculating the visual feature score of the visual block; weighting and fusing the visual feature scores of DOM nodes in the DOM tree with the structural feature scores of the DOM nodes to calculate a comprehensive weight score for the DOM nodes, wherein the visual feature score of the DOM node is the visual feature score of the visual block corresponding to the DOM node; determining the DOM nodes whose comprehensive weight score is greater than a preset weight threshold as container nodes for the main content; and extracting information within the container nodes of the main content to obtain the main information of the webpage.

[0007] Secondly, embodiments of this application also provide a webpage main information extraction device, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a webpage main information extraction method as described in the first aspect above.

[0008] Thirdly, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement a webpage main information extraction method as described in the first aspect above.

[0009] The webpage main information extraction method, device, and medium provided in this application have the following beneficial effects: In this embodiment, a headless browser can be used to obtain the Document Object Model (DOM) tree of a webpage and the visual rendering information of at least one HTML element. Then, based on the visual rendering information, a visual segmentation algorithm is used to divide the webpage into at least one visual block, and the visual feature score of each visual block is calculated. Next, the visual feature scores of DOM nodes in the DOM tree are weighted and fused with the structural feature scores of the DOM nodes to obtain a comprehensive weight score for the DOM nodes. Finally, DOM nodes with a comprehensive weight score greater than a preset weight threshold are determined as container nodes for the main content, thereby extracting information from the container nodes to obtain the main information of the webpage. In this way, the visual layout features of the webpage are deeply fused with the precise structural features of the DOM for extracting the main information. The stability of the visual layout overcomes the variability of the DOM structure, while the precision of the DOM ensures extraction efficiency, achieving high accuracy, robustness, and excellent efficiency in extracting the main information of the webpage. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for extracting main information from a webpage, as provided in this application embodiment; Figure 2 This is a schematic diagram of the internal structure of a webpage main information extraction device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In practical applications, the DOM tree is a tree-like data structure generated by the browser after parsing HTML or XML documents. It's an in-memory representation of the document, mapping each part of the HTML / XML document (tags, attributes, text, comments, etc.) to nodes in the tree. Nodes are connected through parent-child and sibling relationships, forming a hierarchical structure. HTML elements are the basic structural units that make up a webpage. Defined by HTML tags, they describe the content types (such as headings, paragraphs, images, links, etc.) and their layout. Each HTML element consists of a start tag, content, and an end tag (some elements may omit the end tag), and is a core component of the DOM. A headless browser is a webpage processing tool that strips away the graphical user interface (GUI) and runs only in the background. It possesses the core capabilities of a regular browser, such as parsing HTML structure, executing JavaScript code, rendering CSS styles, and handling network requests and cookies, but without displaying visual elements such as windows and address bars.

[0013] This application provides a scheme for extracting the main information of a webpage. The technical solution proposed in this application will be described in detail below with reference to the accompanying drawings.

[0014] Figure 1 This is a flowchart illustrating a method for extracting main information from a webpage, as provided in an embodiment of this application. Figure 1 As shown in the figure, the method for extracting webpage main information provided in this application embodiment specifically includes the following steps: Step 101: Obtain the Document Object Model (DOM) tree of the webpage and the visual rendering information of at least one HTML element through a headless browser.

[0015] In this embodiment, a headless browser can be used to obtain the DOM tree of a webpage and the visual rendering information of at least one HTML element. This automated tool simulates browser behavior, loading the webpage in a graphical interface-free environment, parsing its structure (DOM tree), and extracting the visual rendering information of the HTML elements after rendering. Since the headless browser can execute all JavaScript code, ensuring complete page rendering, it can obtain the complete content actually seen by the user and accurate spatial layout information. Therefore, using a headless browser ensures that all subsequent analyses are based on the page state actually seen by the user, while also ensuring the spatiotemporal consistency of visual and DOM information, providing a reliable data foundation for subsequent cross-modal analysis.

[0016] Compared to purely visual methods that treat the entire page as an image, this application embodiment utilizes only lightweight visual information provided by a headless browser, resulting in less computation and faster processing speed, thus meeting the needs of large-scale industrial data acquisition.

[0017] In practical applications, headless browsers (such as Headless Chrome or Puppeteer) can be used to load web pages, execute all JavaScript code, and complete page rendering. Then, the following key information can be obtained synchronously: first, the complete DOM tree structure; and second, the visual rendering information of each HTML element on the screen (including absolute coordinates, width, height, visibility, etc.). For example, Node.js and the Puppeteer library, or Python libraries such as Playwright, lxml, and BS4Selenium, can be used to obtain the complete DOM tree structure of the web page, where each DOM node contains its precise visual coordinates, size, and visibility information.

[0018] Step 102: Based on the visual rendering information, use a visual segmentation algorithm to divide the webpage into at least one visual block, and calculate the visual feature score of the visual block.

[0019] In this embodiment, based on the aforementioned visual rendering information, a visual segmentation algorithm can be used to divide the webpage into at least one semantically independent visual block. Each visual block represents a logical content area, such as a title block, a body paragraph block, or a navigation bar block. This transforms the unprocessable raw pixel information into structured visual semantic units that can be understood and processed by the algorithm. Furthermore, visual block segmentation helps to directly identify the page's layout structure, including visual cues of content and noise, which aids in subsequently identifying a clear main subject area.

[0020] In this embodiment, a visual feature score can also be calculated for the visual block, which can be used to indicate the importance of the visual block on the webpage. Alternatively, the visual feature score is an indicator that assesses the importance of a visual block on the page by quantifying and analyzing its presentation attributes. Essentially, this uses a mathematical model to simulate the allocation of human visual attention when browsing a webpage. This score transforms human intuitive perception of a page into a computable model, providing an objective basis for the subsequent identification of core content areas.

[0021] In practical applications, visual segmentation algorithms can be improved versions of the VIPS (Vision-based PageSegmentation) algorithm or contour detection methods in computer vision, without specific limitations. The VIPS algorithm is a webpage semantic segmentation algorithm based on visual features, aiming to segment webpages into multiple semantic blocks (such as titles, body text, navigation bars, advertisements, etc.) and construct a hierarchical semantic structure by simulating human visual perception of webpages. The core idea of ​​contour detection algorithms is to extract continuous contour point sets through preprocessing steps such as edge detection and binarization, combined with contour search algorithms (such as chaincode, boundary tracking, convex hull, etc.).

[0022] Step 103: Weight and fuse the visual feature scores of the DOM nodes in the DOM tree with the structural feature scores of the DOM nodes to calculate the comprehensive weight score of the DOM nodes.

[0023] Wherein, the visual feature score of the DOM node is the visual feature score of the visual block corresponding to the DOM node.

[0024] In this embodiment, the visual feature scores of the visual blocks corresponding to a DOM node and the structural feature scores of the DOM node itself can be weighted and fused to obtain a comprehensive weight score for the DOM node. The DOM structural feature score can be used for precise text positioning. This deep fusion of the visual layout features of the webpage and the precise structural features of the DOM allows for mutual verification and inherent error correction capabilities, resisting interference from a single dimension. The stability of the visual layout overcomes the variability of the DOM structure, while the precision of the DOM ensures extraction efficiency. Furthermore, through weight adjustment, it can cover diverse scenarios and has strong adaptability.

[0025] Step 104: Determine the DOM node whose comprehensive weight score is greater than the preset weight threshold as the container node of the main content.

[0026] In this embodiment of the application, the main content can be located based on the comprehensive weight score of the DOM nodes, and the DOM nodes with a comprehensive weight score greater than a preset weight threshold can be identified as the container nodes of the main content.

[0027] In practical applications, the preset weight threshold, as an adjustable parameter, can balance accuracy and robustness. Increasing the preset weight threshold can select a small number of core nodes with extremely high confidence, avoiding the mistaken inclusion of secondary areas (such as related reading lists) in the main body, thus improving accuracy. Moderately decreasing the preset weight threshold can ensure that content with ambiguous boundaries but potential importance is not missed, improving recall and robustness. Dynamic adjustments can be made based on website type (such as news, forums, encyclopedias) or task requirements (such as strictly extracting the main text vs. leniently including the comment section), allowing the same algorithm to flexibly adapt to different scenarios without modifying the core model. This results in a high degree of automation and reduced maintenance costs. Moreover, subsequent content extraction only needs to be performed within the aforementioned container nodes, greatly narrowing the search scope and improving processing speed.

[0028] Through the above steps, even if a website's front-end redesign causes a major change in the DOM structure, as long as the main content of the page maintains its general visual layout and position, it can still be accurately located using visual features. This overcomes the fatal flaw of pure DOM analysis methods, which "always fail with redesigns." Moreover, through dual filtering and verification using visual features (excluding noise such as advertisements and navigation) and DOM structure features (precise text positioning), the true main content can be identified with extreme accuracy, effectively reducing the false extraction rate.

[0029] Step 105: Extract information from the container node of the main content to obtain the main information of the webpage.

[0030] In this embodiment, after locating the container node of the main content, information can be extracted within it. Compared to extracting from the entire page, this method is almost entirely free of irrelevant noise, resulting in extremely high purity. It also improves processing efficiency and speed. Furthermore, the container node defines a natural semantic boundary within the DOM tree. Text, paragraphs, images, etc., within this boundary typically belong to the same logical theme and narrative flow. Extracting information within the container node ensures the semantic coherence of the main information. This ensures the quality of the webpage's main information and the efficiency of its extraction, thereby providing high-quality raw materials for subsequent user processing of the webpage's main information (e.g., direct reading, analysis, or storage).

[0031] In this embodiment, a headless browser can be used to obtain the Document Object Model (DOM) tree of a webpage and the visual rendering information of at least one HTML element. Then, based on the visual rendering information, a visual segmentation algorithm is used to divide the webpage into at least one visual block, and the visual feature score of each visual block is calculated. Next, the visual feature scores of DOM nodes in the DOM tree are weighted and fused with the structural feature scores of the DOM nodes to obtain a comprehensive weight score for the DOM nodes. Finally, DOM nodes with a comprehensive weight score greater than a preset weight threshold are determined as container nodes for the main content, thereby extracting information from the container nodes to obtain the main information of the webpage. In this way, the visual layout features of the webpage are deeply fused with the precise structural features of the DOM for extracting the main information. The stability of the visual layout overcomes the variability of the DOM structure, while the precision of the DOM ensures extraction efficiency, achieving high accuracy, high robustness, and excellent efficiency in extracting the main information of the webpage.

[0032] In practical applications, the webpage main information extraction method provided in this application embodiment does not require writing specific rules for different websites. It is a general solution with a high degree of automation and reduced maintenance costs.

[0033] In one possible implementation, before dividing the webpage into at least one visual block using a visual segmentation algorithm based on the visual rendering information, the method further includes: Filter out invisible elements from the DOM tree; The browser viewport of the webpage is defined as an initial visual block.

[0034] In the above embodiment, all invisible elements with a `rect` value of `null` can be filtered out from the DOM tree. In practical applications, elements with a `rect` value of `null` typically mean `display: none`, are out of the document flow and not rendered, or have a size of 0. They are absolutely visually invalid elements (such as hidden script templates or CSS pseudo-element placeholders). Pre-filtering them is equivalent to cleaning the data from the source, reducing noise interference, improving subsequent processing efficiency, and reducing computational complexity. Then, the entire browser viewport of the webpage can be used as an initial visual block. Logically, when a user opens a webpage, the first thing they see is the entire visible area (Viewport). This is the top-level, undivided visual unit. Algorithmically, it provides a suitable starting point for subsequent visual block division. Thus, in subsequent processes, a visual segmentation algorithm can be used to divide this initial large block into smaller semantic blocks, thereby simplifying the logic of the initial visual block division.

[0035] In one possible implementation, dividing the webpage into at least one visual block using a visual segmentation algorithm based on the visual rendering information includes: Based on the visual rendering information, the boundaries of the child elements inside the current visual block are determined, wherein the child elements are the elements actually displayed in the DOM tree; Based on the boundaries of the sub-elements, candidate separators for the visual block are determined in the vertical and horizontal directions; If the intensity of the candidate separator is greater than a preset intensity threshold, the candidate separator is determined as the target separator, wherein the intensity is determined based on the difference in the width and style of the blank area between adjacent child elements; The current visual block is divided into two sub-blocks based on the target separator, wherein the sub-blocks are the new visual blocks; Repeat the above steps for the sub-block until there are no target separators in the visual block.

[0036] In the above embodiment, for the current visual block, the boundaries of all its child elements (e.g., text, images, divs, etc.) can be found based on visual rendering information. Then, based on the boundaries of these child elements, candidate separators for the visual block in the vertical and horizontal directions are determined. That is, based on the left and right boundaries of the child elements, possible candidate separators in the vertical direction are determined, and based on the top and bottom boundaries of the child elements, possible candidate separators in the horizontal direction are determined. Subsequently, the strength of each candidate separator can be evaluated one by one. The strength is determined by the width (or height) of the blank area between adjacent child elements and the difference in style between adjacent elements (e.g., background color, border). If the strength of a candidate separator is greater than a preset strength threshold, the candidate separator is determined as the target separator. For example, if there is a candidate separator with a blank width > 10px and a background color difference > 10%, it can be determined as the target separator. Then, the current visual block is divided into two sub-blocks along the target separator. For each newly generated sub-block, the above steps are repeated until it can no longer be divided (i.e., the elements within the block are compactly laid out with no obvious separators). In this way, it does not rely on the DOM structure, but is based on the visual rendering result, unaffected by the complexity of HTML nesting, and has strong robustness. It can find reasonable segmentation under different screen sizes, adapt to responsive design, and can be recognized as soon as it is rendered, and can handle dynamically generated content. Moreover, by setting a preset intensity threshold, it can maintain visual integrity without destroying semantic units.

[0037] In one possible implementation, the visual feature score includes a visual importance score and a content density score; The calculation of the visual feature score of the visual block includes: Based on the area ratio and center position ratio of the visual block, the visual importance score of the visual block is determined, wherein the visual importance score is used to indicate the importance of the visual block in the user's field of vision; The content density score of the visual block is determined based on the ratio of the area occupied by the text node of the visual block to the area of ​​the visual block, wherein the content density score is used to characterize the density of text information within the visual block.

[0038] In the above embodiments, the visual importance score can characterize the importance of the visual block in the user's field of vision. When calculating, both the center position and area of ​​the visual block need to be considered. Generally, areas located in the center of the browser viewport and with a larger area are more likely to be the main content. For example, the score is directly proportional to the area and inversely proportional to the distance from the center point to the top.

[0039] In practical applications, a weighted function based on location and area can be used to determine the visual importance score. iFor example, the formula is as follows: Simportance i = (Area i / Total Area ) * W area + F_centrality i * W centrality Among them, Area i For visual block V i Area (width * height), Total Area F_centrality represents the total area of ​​all visual blocks on the entire page. i The centrality function calculates the reciprocal of the normalized distance from the visual block center (Cx_i, Cy_i) to the viewport center (Vw / 2, Vh / 2).

[0040] F_centrality i = 1 / (1 + sqrt( (Cx_i - Vw / 2) 2 + (Cy_i - Vh / 2) 2 ) / D max ) D max It is one-third of the length of the viewport diagonal, used for normalization.

[0041] It should be noted that W area and W centrality The weighting coefficients can be optimized experimentally; for example, they can be set to W. area = 0.6, W centrality = 0.4.

[0042] In the above embodiments, the content density score Sdensity i This can be used to characterize the importance of a visual block in the user's field of vision. In practical applications, the content density score can be determined by calculating the ratio of the area occupied by plain text content within the visual block to the total area of ​​the visual block. The main content area is typically text-dense, while navigation bars or advertising areas have a higher proportion of links or images. The higher this ratio, the higher the score. An example calculation formula is as follows: Sdensity i = Text_Area i / Area i Among them, Text_Area i For visual block V iThe approximate total area occupied by all text nodes within the text area. This can be approximated by estimating: total text length * average character width * line height, or more precisely, by summing the areas of each text node calculated based on its font size. A higher ratio indicates a greater likelihood that the area contains narrative text.

[0043] In practical applications, the visual feature scores of visual blocks can also be calculated directly, without any specific restrictions.

[0044] In one possible implementation, the step of weightedly fusing the visual feature scores and structural feature scores of DOM nodes in the DOM tree to calculate the comprehensive weight score of the DOM nodes includes: Based on the overlap between the visual rectangle of the DOM node in the DOM tree and the visual block, establish the mapping relationship between the visual block and the DOM node; Calculate the structural feature score of the DOM node based on its structural features; The visual feature score corresponding to the DOM node is weighted and fused with the structural feature score of the DOM node to obtain the comprehensive weight score of the DOM node.

[0045] In practical applications, a visual block V can be established. i To DOM node N j The mapping relationship. A visual block typically corresponds to one or more DOM nodes (usually block-level elements such as...). , <section>Mapping can be established by determining the overlap (IoU) between the visual rectangle of a DOM node and the visual block. IoU = intersection area of ​​the visual block and the DOM node ÷ union area. A threshold can then be applied, and mapping is established if IoU > the threshold (e.g., 0.7). This approach provides a clear geometric relationship, is independent of any specific DOM structure, and is computationally simple.

[0046] Then, based on the structural features of DOM nodes, the structural feature score (Sstructure) of DOM nodes is calculated. j Subsequently, the visual feature score of the DOM node is weighted and fused with the structural feature score of the DOM node to obtain the comprehensive weight score of the DOM node.

[0047] For example, for DOM node N j Its total weighted score (Stotal) j The formula for calculating ) is: Stotal j = α * (Simportance i + Sdensity i ) / 2 + β * Sstructure j Here, α and β are fusion coefficients, and α + β = 1. Optimization on the validation set can be achieved through grid search. Typical values ​​such as α = 0.7 and β = 0.3 indicate a greater emphasis on visual features, adhering to the "what you see is what you get" principle—what the user sees is what matters. Simultaneously, structural correction can rectify visual segmentation errors and provide semantic enhancement. In this way, by combining layout and structural information, a multi-dimensional evaluation is performed to obtain the comprehensive weight score of the DOM node. Furthermore, there is a backup even if a single feature fails, demonstrating good robustness. Moreover, the above calculation steps can improve computational efficiency.

[0048] In one possible implementation, calculating the structural feature score of the DOM node based on its structural features includes: The label score of the DOM node is determined based on the label of the DOM node and the predefined weight dictionary; The semantic score of the DOM node is determined by matching the node class name and ID of the DOM node using regular expressions. The link text ratio of the DOM node is determined by the ratio of the text length of the hyperlink tag of the DOM node to the total text length of the DOM node; The structural feature score of the DOM node is determined based on the tag score, semantic score, and link text ratio.

[0049] In practical applications, for example, the structural feature score (Sstructure) of a DOM node. j The calculation formula is as follows: Sstructure j = W tag * Score tag (N j ) + W semantic * Score semantic (N j ) + W link * (1 -Link_Ratio j ) Among them, the label score tag The weight is determined based on a predefined label weight dictionary. It can be assigned... <article> , , Tags with higher weight are assigned <script>,<style>,<nav>等标签较低或负权重。例如： {'ARTICLE': 1.0, 'SECTION': 0.8, 'P': 0.9, 'DIV': 0.5, 'SPAN': 0.2, 'NAV': -0.5, 'SCRIPT': -1.0, 'STYLE': -1.0}。

[0050] 语义分数 (Scoresemantic）可以基于节点类名和ID的正则表达式匹配确定。在实际应用中，可以分析DOM节点的类名（class）和ID，若包含如content, article, post-body等关键词，则赋予较高权重；若包含ad, footer, sidebar等关键词，则降低权重。

[0051] 例如，正向关键词（如 / content|article|post|main|body / i）：匹配到则+1.0。负向关键词（如 / ad|banner|footer|sidebar|menu|comment / i）：匹配到则-1.0。

[0052] 在实际应用中，链接文本比可以通过计算节点内所有标签的文本长度占节点总文本长度的比例来获得。比例过高，表明该区域可能是导航或推荐列表，而非主体正文，故降低其权重。

[0053] 在实际应用中，链接文本比 (Link_Ratioj)可以通过以下公式进行计算：Link_Ratioj= (DOM节点Nj内所有标签的文本长度之和) / (Nj节点的总文本长度)。这个值越高，说明NOM节点是导航或广告的可能性越大，故用 (1 - Link_Ratio_j)作为权重因子。

[0054] 需要说明的是，Wtag, Wsemantic, Wlink为权重，例如 0.4, 0.3, 0.3。

[0055] 在实际应用中，也可以通过其他方式计算DOM节点的结构特征分数，例如，只考虑标签分数和语义分数，具体不作限制。

[0056] 在一个可能的实现方式中，所述确定所述综合权重分数大于预设权重阈值的所述DOM节点为主体内容的容器节点，包括：采用自底向上的贪心算法遍历所述DOM树；在目标比值大于预设比值阈值，且父节点的面积大于预设面积阈值的情况下，将所述父节点确定为主体内容的容器节点，其中，所述目标比值为所述父节点的综合权重分数与所述父节点的子节点的综合权重分数之和之间的比值。

[0057] 在实际应用中，可以采用一种"自底向上，投票聚合”的策略遍历DOM树。即可以采用自底向上的贪心算法遍历DOM树。从叶子节点开始，向上聚合。可以计算父节点的综合权重分数Stotalparent与其直接子节点的综合权重分数Stotalchildren之和的目标比值Ratio：Ratio = Stotalparent / Σ(Stotalchildren)。

[0058] 如果同时满足：Ratio>λ（λ为一个预设比值阈值，例如1.5），说明父节点作为一个容器的凝聚力很强，且父节点对应的视觉块面积超过预设面积阈值（例如视口面积的20%），则该父节点可以被确定为主体内容的容器节点 Nmain。

[0059] 需要说明的是，由于是自底向上的贪心算法，通常会在找到第一个（即最深的）满足条件的父节点时就将其作为主体容器，并停止向上遍历。这是因为自底向上遍历，先访问的是较低的父节点，然后逐渐向上。如果较低的父节点已经满足条件，那么它很可能是最内层的内容容器，这样提取的内容更精确，不会包含过多无关内容。如果继续向上，可能会找到一个更大的容器，但可能包含更多非内容部分（如侧边栏、评论等），导致内容不纯。

[0060] 在实际应用中，网页的主体内容通常包含在较大的容器中，并且该容器在视觉上和结构上都是子节点的聚合。自底向上可以从较小的内容节点开始，逐步向上寻找能够容纳这些内容的较大容器，从而找到最合适的容器节点。如果采用自顶向下的方式，可能会过早地选择到过大的容器（例如整个body），而自底向上可以确保选择的容器足够具体，因为它是从实际内容节点开始向上寻找，直到满足条件的第一个父节点。而且不同网站的结构差异很大，自底向上的贪心算法不依赖于固定的标签或类名，而是根据节点的综合权重分数和面积进行判断，因此能够适应各种不同的网页设计、适应性强。而且即使部分叶子节点识别有误，上层节点的综合评估仍能保持稳定，且贪心算法结合阈值过滤，可以平衡计算速度和精度，鲁棒性好。

[0061] 在一个可能的实现方式中，所述在所述主体内容的容器节点内进行信息提取，获取所述网页的主体信息，包括：基于所述容器节点内的<h1>或<h2>标签文本、或字体加粗的文本块，获得标题；通过遍历所述容器节点内所有的文本节点，根据段落标签或视觉块分隔进行分段，获得正文；使用预定义的正则表达式模式，在所述容器节点内及对应的兄弟节点中搜索匹配时间日期格式的文本，获得发布时间。

[0062] 在上述实施例中，可以在 Nmain内进行操作，进行标题、正文以及发布时间的提取。在实际应用中，在进行标题提取时，可以选取容器内权重最高的<h1>或<h2>标签文本，或字体最大、加粗的文本块。例如，可以优先查找 Nmain内或其紧前兄弟节点中的第一个<h1>标签。若无，则选择字体大小为最大、且加粗的文本节点。也可以通过遍历容器节点的所有子节点，找到所有<h1>和<h2>标签，以及所有文本节点，计算其权重（比如根据字体大小、加粗等属性），然后选择权重最高的作为标题，具体不作限制。这样，通过多种策略（标签、样式）确保找到正确的标题，可以提高标题提取的召回率和准确率。

[0063] 在进行正文提取时，可以遍历容器内所有文本节点，根据段落标签（）和视觉块分隔进行分段，清理空白字符，合并成连贯正文。在实际应用中，可以遍历 Nmain下所有文本节点，根据其块级父标签（如）或视觉块分隔（比如通过CSS样式判断是否为块级元素）进行自然分段。使用 Readability类似的算法清理无关字符，合并成连贯文本。这样，通过结合HTML标签和视觉块分隔，能够更准确地划分段落，保留原文的结构和顺序，同时去除噪声，得到高质量的正文。

[0064] 在进行发布时间提取时，可以使用预定义的正则表达式模式，在容器内或其兄弟节点中搜索匹配时间日期格式的文本。例如，可以使用一组强大的正则表达式在 N_main及其祖先节点（向上2层）的文本中进行匹配。

[0065] 示例性的，正则表达式如下：r'(\d{4}[-年 / ]\d{1,2}[-月 / ]\d{1,2}\s*\d{1,2}:\d{1,2}:\d{1,2})|(\d{1,2}[-月 / ]\ d{1,2}[-月 / ]\d{4})|(发布时间[:：]\s*[\d-]+)'这样，通过正则表达式在容器及其周围搜索，能够适应不同网站的时间格式，提高时间提取的成功率。

[0066] 在上述实施例中，在已经定位到主体容器的基础上进行精细提取，避免了对整个DOM树的遍历，提高了提取速度，也减少了无关信息的干扰。整体流程具备高效性。而且，每个步骤相对独立，可以根据需要调整或替换其中的策略，例如增加新的时间格式正则表达式，或调整正文分段的逻辑，可维护性和扩展性较高。

[0067] 以上为本申请提出的方法实施例。基于同样的发明构思，本申请实施例还提供了一种网页主体信息提取设备，其结构如图2所示。

[0068] 图2为本申请实施例提供的一种网页主体信息提取设备内部结构示意图。如图2所示，设备包括：至少一个处理器201；以及，与至少一个处理器通信连接的存储器202；其中，存储器202存储有可被至少一个处理器执行的指令，指令被至少一个处理器201执行，以使至少一个处理器201能够：执行上述网页主体信息提取方法。

[0069] 在本申请实施例的一种或多种可能实现方式中，前述处理器用于，通过无头浏览器获取网页的文档对象模型DOM树及至少一个HTML元素的视觉渲染信息；基于所述视觉渲染信息，使用视觉分隔算法将所述网页划分为至少一个视觉块，并计算所述视觉块的视觉特征分数；将所述DOM树中DOM节点的视觉特征分数与所述DOM节点的结构特征分数进行加权融合，计算所述DOM节点的综合权重分数，其中，所述DOM节点的视觉特征分数为所述DOM节点对应的视觉块的视觉特征分数；确定所述综合权重分数大于预设权重阈值的所述DOM节点为主体内容的容器节点；在所述主体内容的容器节点内进行信息提取，获取网页的主体信息。

[0070] 本申请的一些实施例提供的对应于图1的一种的非易失性计算机存储介质，存储有计算机可执行指令，计算机可执行指令设置为：执行上述网页主体信息提取方法。

[0071] 在本申请实施例的一种或多种可能实现方式中，前述计算机可执行指令设置为能够执行，通过无头浏览器获取网页的文档对象模型DOM树及至少一个HTML元素的视觉渲染信息；基于所述视觉渲染信息，使用视觉分隔算法将所述网页划分为至少一个视觉块，并计算所述视觉块的视觉特征分数；将所述DOM树中DOM节点的视觉特征分数与所述DOM节点的结构特征分数进行加权融合，计算所述DOM节点的综合权重分数，其中，所述DOM节点的视觉特征分数为所述DOM节点对应的视觉块的视觉特征分数；确定所述综合权重分数大于预设权重阈值的所述DOM节点为主体内容的容器节点；在所述主体内容的容器节点内进行信息提取，获取网页的主体信息。

[0072] 本申请中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于物联网设备和介质实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。

[0073] 本申请实施例提供的系统和介质与方法是一一对应的，因此，系统和介质也具有与其对应的方法类似的有益技术效果，由于上面已经对方法的有益技术效果进行了详细说明，因此，这里不再赘述系统和介质的有益技术效果。

[0074] 本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质（包括但不限于磁盘存储器、CD-ROM、光学存储器等）上实施的计算机程序产品的形式。

[0075] 本申请是参照根据本申请实施例的方法、设备（系统）、和计算机程序产品的流程图和／或方框图来描述的。应理解可由计算机程序指令实现流程图和／或方框图中的每一流程和／或方框、以及流程图和／或方框图中的流程和／或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的装置。

[0076] 这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能。

[0077] 这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的步骤。

[0078] 在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入 / 输出接口、网络接口和内存。

[0079] 内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和 / 或非易失性内存等形式，如只读存储器(ROM)或闪存(flashRAM)。内存是计算机可读介质的示例。

[0080] 计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitorymedia)，如调制的数据信号和载波。

[0081] 还需要说明的是，术语"包括”、"包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。

[0082] 以上所述仅为本申请的实施例而已，并不用于限制本申请。对于本领域技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本申请的权利要求范围之内。< / script> < / article> < / section>

Claims

1. A method for extracting main information from a webpage, characterized in that, include: Obtain the Document Object Model (DOM) tree of a webpage and the visual rendering information of at least one HTML element through a headless browser. Based on the visual rendering information, the webpage is divided into at least one visual block using a visual segmentation algorithm, and the visual feature score of the visual block is calculated. The visual feature scores of DOM nodes in the DOM tree are weighted and fused with the structural feature scores of the DOM nodes to calculate the comprehensive weight score of the DOM nodes, wherein the visual feature score of the DOM nodes is the visual feature score of the visual block corresponding to the DOM nodes; The DOM nodes whose comprehensive weight score is greater than a preset weight threshold are identified as container nodes for the main content; Information is extracted within the container node of the main content to obtain the main information of the webpage.

2. The method according to claim 1, characterized in that, Before dividing the webpage into at least one visual block using a visual segmentation algorithm based on the visual rendering information, the method further includes: Filter out invisible elements from the DOM tree; The browser viewport of the webpage is defined as an initial visual block.

3. The method according to claim 2, characterized in that, The step of dividing the webpage into at least one visual block using a visual segmentation algorithm based on the visual rendering information includes: Based on the visual rendering information, the boundaries of the child elements inside the current visual block are determined, wherein the child elements are the elements actually displayed in the DOM tree; Based on the boundaries of the sub-elements, candidate separators for the visual block are determined in the vertical and horizontal directions; If the intensity of the candidate separator is greater than a preset intensity threshold, the candidate separator is determined as the target separator, wherein the intensity is determined based on the difference in the width and style of the blank area between adjacent child elements; The current visual block is divided into two sub-blocks based on the target separator, wherein the sub-blocks are the new visual blocks; Repeat the above steps for the sub-block until there are no target separators in the visual block.

4. The method according to claim 1, characterized in that, The visual feature score includes a visual importance score and a content density score; The calculation of the visual feature score of the visual block includes: Based on the area ratio and center position ratio of the visual block, the visual importance score of the visual block is determined, wherein the visual importance score is used to indicate the importance of the visual block in the user's field of vision; The content density score of the visual block is determined based on the ratio of the area occupied by the text node of the visual block to the area of ​​the visual block, wherein the content density score is used to characterize the density of text information within the visual block.

5. The method according to claim 1, characterized in that, The step of weighting and fusing the visual feature scores and structural feature scores of DOM nodes in the DOM tree to calculate the comprehensive weight score of the DOM nodes includes: Based on the overlap between the visual rectangle of the DOM node in the DOM tree and the visual block, establish the mapping relationship between the visual block and the DOM node; Calculate the structural feature score of the DOM node based on its structural features; The visual feature score and the structural feature score of the DOM node are weighted and fused to obtain the comprehensive weight score of the DOM node.

6. The method according to claim 5, characterized in that, The calculation of the structural feature score of the DOM node based on its structural features includes: The label score of the DOM node is determined based on the label of the DOM node and the predefined weight dictionary; The semantic score of the DOM node is determined by matching the node class name and ID of the DOM node using regular expressions. The link text ratio of the DOM node is determined by the ratio of the text length of the hyperlink tag of the DOM node to the total text length of the DOM node; The structural feature score of the DOM node is determined based on the tag score, semantic score, and link text ratio.

7. The method according to claim 1, characterized in that, The step of determining the DOM node whose comprehensive weight score is greater than a preset weight threshold as the container node of the main content includes: A bottom-up greedy algorithm is used to traverse the DOM tree; If the target ratio is greater than a preset ratio threshold and the area of ​​the parent node is greater than a preset area threshold, the parent node is determined as the container node of the main content. The target ratio is the ratio between the comprehensive weight score of the parent node and the sum of the comprehensive weight scores of the child nodes of the parent node.

8. The method according to claim 1, characterized in that, The step of extracting information within the container node of the main content to obtain the main information of the webpage includes: Based on the container node <h1> or< / h1> <h2> Obtain the title from the tag text or the text block in bold;< / h2> By traversing all text nodes within the container node, the text is segmented according to paragraph tags or visual blocks to obtain the main text. Using a predefined regular expression pattern, search for text matching the date and time format within the container node and its corresponding sibling nodes to obtain the publication time.

9. A device for extracting main information from a webpage, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a webpage main information extraction method as described in any one of claims 1-8.

10. A computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, a webpage main information extraction method as described in any one of claims 1-8 is implemented.

Images