Webpage content extraction method based on rule and statistical feature combination

Abstract

The invention relates to a webpage content extraction method based on rule and statistical feature combination, which comprises the following steps of: converting a webpage html source code into a dom tree, reserving tree nodes which are identified as a text area, cutting tree nodes which are identified as a noise area, and performing copy storage on the dom tree of which the noise area is removed; and based on a combined switching mode of rule extraction and statistical feature extraction, extracting content from the stored dom tree copy. Compared with the prior art, the quality and efficiency of webpage content extraction can be improved, the accuracy and comprehensiveness of the extracted content are enhanced, and the webpage data extraction processing speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of web page extraction, and in particular to a web page content extraction method based on a combination of rules and statistical features. Background Art

[0002] In most cases, web page extraction refers to extracting structured data from web pages. However, existing web page extraction technologies face multiple technical bottlenecks when processing large-scale and diverse web page data (such as CommonCrawl), mainly including:

[0003] 1. Limited ability to extract diverse web page data: Current general web page extraction methods often fail to fully cover all types of web pages (such as news, forums, Q&A, etc.), resulting in significant defects in the quality of unconventional web page data extraction. For example, atypical reading pages such as forums and Q&A have missing core texts and retention issues due to the presence of interlaced text and images, text and image pairs, list clusters, navigation bars, etc., which seriously affects the effectiveness of subsequent data processing and analysis.

[0004] 2. The lack of web page content cannot guarantee integrity: Current web page extraction methods mainly retain text content, but often ignore elements such as pictures, code blocks and paragraphs, resulting in incomplete information, loss of context, data analysis bias, impaired user experience and other problems, greatly reducing the quality of multimodal data.

[0005] 3. Poor quality of web page extraction: Existing web page extraction technology extracts mixed content. In addition to the main text, it also contains advertisements, comments, js, css and other noise area content, which cannot effectively obtain high-quality data.

[0006] 4. Insufficient multi-language extraction capabilities: Existing web page extraction technologies usually only have the ability to extract Chinese and English texts. For extraction of other languages, it cannot guarantee the format and structure of the text to be adapted, resulting in poor quality of web page content in languages ​​other than Chinese and English, which cannot be used for model training.

[0007] 5. The format of web page formulas cannot be standardized: Current web page extraction technology cannot convert mathematical and chemical formulas on web pages into a unified standard latex format, which affects the readability and reusability of the formulas in the data and limits the subsequent processing and analysis of the data.

[0008] 6. Low web data extraction speed: Existing web extraction technology has a low data extraction speed. It takes 1-2 months to extract 250 billion CommonCrawl data. It has weak ability to process large amounts of data and is time-consuming, which is not conducive to efficient data acquisition and rapid iteration.

[0009] In addition, the prior art also proposes to use justText and Web2Text algorithms for web page extraction, among which jusText is an algorithm for extracting text content from web pages. Its goal is to extract meaningful text content from web pages containing a lot of messy information, while filtering out irrelevant information such as advertisements, navigation bars, and sidebars. The core idea of ​​jusText is to identify the elements that are most likely to contain text content by analyzing the structure of the web page. It uses a scoring-based method to evaluate the importance of each HTML element, and then selects the element with the highest score as the text content.

[0010] Web2Text usually uses a scoring-based approach to evaluate the importance of each HTML element, analyzes the structure of the web page, and identifies the elements that are most likely to contain the body content. The core method is to regard extraction as a sequence tagging task, divide the HTML into several blocks, and feed the structural features from each block and its adjacent blocks to CNN. A hidden Markov model is used on top of the CNN-based block representation to select the element with the highest score as the body content.

[0011] However, for the two algorithms mentioned above, jusText cannot accurately extract all relevant content for web pages containing complex layouts, such as tables and lists. jusText mainly focuses on text content, so it cannot effectively process non-text elements such as images and videos. For web pages containing mathematical formulas or code blocks, jusText cannot retain these special formats. When the website structure changes, the content extracted by jusText may lack certain key information. Summary of the invention

[0012] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a web page content extraction method based on a combination of rules and statistical features, which can improve the quality and efficiency of web page content extraction, enhance the accuracy and comprehensiveness of the extracted content, and increase the processing speed.

[0013] The object of the present invention can be achieved by the following technical solution: A web page content extraction method based on a combination of rules and statistical features, comprising the following steps:

[0014] S1. Convert the HTML source code of the web page into a DOM (document object model) tree, retain the tree nodes identified as the text area, cut the tree nodes identified as the noise area, and save a copy of the DOM tree with the noise area removed;

[0015] S2. Extract content from the saved DOM tree copy based on the combined switching method of rule extraction and statistical feature extraction.

[0016] Furthermore, the step S1 includes the following steps:

[0017] S11, converting the web page HTML source code into a DOM tree, and then performing tag formatting and mathematical formula conversion processing;

[0018] S12, for the DOM tree after the tag formatting and mathematical formula conversion processing, performing noise area clipping based on HTML structure and noise area clipping based on style features respectively;

[0019] S13, saving a copy of the DOM tree with the noise area removed after cropping.

[0020] Furthermore, the process of label formatting and mathematical formula conversion processing in step S11 includes:

[0021] When the alt attribute of the image tag contains latex, a span node is inserted before the img tag node, and the text content is assigned to the img alt attribute content;

[0022] When the type of the script tag is "math / tex" or "math / asciimath", convert the script tag to a span tag, and assign the text content to the text content of the script;

[0023] When the tag is math, a local tag mapping table is constructed to convert the formula written in XML to standard latex, convert the math tag to a span tag, and assign the text content to the converted standard latex.

[0024] When the tag is mathjax, the asciimath2tex function is implemented to convert it into standard latex, convert the mathjax tag into a span tag, and assign the text content to the converted standard latex.

[0025] Furthermore, the HTML structure-based noise region cropping in step S12 is specifically to automatically identify and remove non-main content regions in the web page, including list clusters and navigation bars, by calculating the HTML tag set and text tag density.

[0026] Furthermore, the process of cutting out the noise area based on the HTML structure in step S12 includes:

[0027] For entries with multiple common parent nodes, where multiple entries constitute a list cluster, the labels for constructing the list cluster include li, div, ul, and section. When a common parent node contains multiple parallel child nodes and each child node contains a link a label, when the number of child nodes satisfies>=5, the labels and attribute names of the child nodes, the labels and attribute names in the grandchild nodes are concatenated into string features, and the Jaccard similarity of the string features of each pair of child nodes is calculated. The lowest value is removed, and the average of the remaining values ​​is the possibility that the current area is a list area. When this value is <0.85, it is regarded as a list area and is removed.

[0028] Furthermore, the noise region cropping based on style features in step S12 specifically generates noise region labels, class, and id keyword tables through statistical analysis, and identifies and removes non-essential content in the web page, including advertisements, comments, JavaScript, and CSS.

[0029] Furthermore, the process of cutting out the noise region based on the style feature in step S12 includes:

[0030] Statistical analysis of the top 1000 popular sites is performed to generate a noise area tag, class, and id keyword table. The noise area style features include: style="display:none", class="advertisement", id="paywall". When an HTML tag attribute hits the above keyword table, it is scored based on the length of the text in the current tag, link density, and stop word density. When the score is lower than the preset score threshold, the current area is determined to be a noise area and is removed.

[0031] Furthermore, the combined switching process based on rule extraction and statistical feature extraction in step S2 includes:

[0032] Count the tags, classes, and ids in the HTML text area of ​​the top 1000 popular sites, and build a text rule table. The text area rules include: class="article", id="content", class="post-text";

[0033] When an HTML is input, after HTML tag standardization and noise clipping, it is first matched based on the text rules. When the extraction result does not trigger the list noise clipping, if the current result non-a tag text length is > 20 characters, the extraction result of the web page body area content is directly returned; if the current result non-a tag text length is <= 20 characters and the non-a tag text length in the p tag of the entire HTML is > 100 characters, then switch to the extraction method based on statistical features to extract the web page body area content;

[0034] When the extraction result triggers list noise trimming, if the current result non-a tag text length is > 50 characters, the extraction result of the web page body area content is directly returned; if the current result non-a tag text length is <= 50 characters and the non-a tag text length in the p tag in the entire HTML is > 100 characters, then switch to the statistical feature-based extraction method to extract the web page body area content.

[0035] Furthermore, the specific process of extracting the content of the web page body area using the statistical feature-based extraction method is as follows:

[0036] First, divide the content into blocks: analyze the properties of the nodes and the relationship between the upper and lower nodes, and identify the content blocks containing the main text;

[0037] Then, score the content blocks: assign a corresponding score to each content block;

[0038] Finally, select the best content block: select the content block with the highest score, which contains the main content of the article.

[0039] Furthermore, the content block score is specifically calculated based on the length of the text in the block, the link density, the stop word density, and the appearance of specific tags. , <article>, it is considered a positive signal and is given extra points. <nav> , <aside>, then points will be deducted.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] The present invention first converts the HTML source code of a web page into a DOM tree, retains the tree nodes identified as the text area, cuts the tree nodes identified as the noise area, and saves a copy of the DOM tree without the noise area; then extracts content from the saved DOM tree copy based on a combined switching method of rule extraction and statistical feature extraction. After converting the HTML source code of a web page into a DOM tree, the label formatting and mathematical formula conversion processing are first performed, so that the mathematical formulas in the web page can be converted into a standard LaTeX format to avoid being misprocessed when cutting the noise area later;

[0042] When performing noise region clipping, the present invention adds noise region clipping based on HTML structure, which can identify and remove non-main content areas in web pages, such as list clusters, navigation bars, etc., thereby reducing noise interference and improving content quality; on the other hand, optimizes noise region clipping based on style features, which can identify and remove non-main content such as advertisements, comments, JavaScript and CSS in web pages, thereby improving the purity of extracted content;

[0043] In addition, the present invention adopts a combined switching method based on rule extraction and statistical feature extraction, determines the extraction method through threshold condition judgment, improves the text extraction accuracy by combining the two methods based on rule extraction and statistical feature extraction, and designs to include text content length judgment in the threshold condition judgment. When the threshold judgment conditions are consistent, the main text area will be directly identified after the HTML is converted into a DOM tree, which is not affected by the language type and has strong compatibility and versatility for multilingual content.

[0044] In the present invention, tree nodes identified as text areas are retained, and tree nodes identified as noise areas are pruned, and a copy of the DOM tree with the noise area removed is saved. When the result threshold of the subsequent rule-based extraction method does not meet the requirements and is switched to an extraction method based on statistical features, the text area is re-identified from the DOM tree copy, and finally the DOM tree is reconverted into HTML source code, which can completely retain the original format and information of elements such as pictures, paragraphs, code blocks, etc. in the web page, ensuring that the extracted content remains highly consistent with the original web page text area after rendering.

[0045] The present invention reconstructs the processing flow, removes unnecessary steps, and no longer processes and generates results one by one based on a single HTML node, while avoiding the problem of multimodal image loss. The present invention advances the operation of cutting noise nodes, greatly reducing the number of nodes judged and processed by the calculation and scoring of subsequent text area recognition, and reducing the complexity of the algorithm. The improved process is more concise and efficient, the shortest path is to directly apply the rule (rule) for content extraction, and the longest path is only for the rule (rule) to transfer to the statistical feature for content extraction, which greatly improves the processing speed, can achieve rapid extraction of web page content, and does not lose the integrity of the content while ensuring the processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 It is a schematic diagram of the application process of the embodiment;

[0048] Figure 3 Schematic diagram of retaining and cutting nodes after converting HTML to DOM tree;

[0049] Figure 4 A schematic diagram of the process of combining and switching between rule-based extraction and statistical feature extraction;

[0050] Figure 5 Schematic diagram for list cluster identification. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example

[0053] like Figure 1 As shown, a web page content extraction method based on a combination of rules and statistical features includes the following steps:

[0054] S1. Convert the HTML source code of the web page into a DOM tree, retain the tree nodes identified as the text area, cut the tree nodes identified as the noise area, and save a copy of the DOM tree with the noise area removed;

[0055] S2. Extract content from the saved DOM tree copy based on the combined switching method of rule extraction and statistical feature extraction.

[0056] Applying the above solution to practice, Figure 2 As shown, the main contents are:

[0057] 1. Keep the source code of the main area webpage:

[0058] Many existing extraction methods will destroy the original web page structure and only output the plain text content. This scheme designs an extraction method by converting the HTML source code into a DOM tree, in which the tree nodes identified as the text area are retained and the tree nodes identified as the noise area are cropped (such as Figure 3 As shown in the figure, a copy of the DOM tree with the noise area removed is saved at the same time. When the result threshold of the subsequent rule-based extraction method does not meet the requirements and is switched to the statistical feature-based extraction method, the body area is re-identified from the DOM tree copy, and finally the DOM tree is converted back into HTML source code, which can completely retain the original format and information of elements such as pictures, paragraphs, and code blocks in the web page, ensuring that the extracted content remains highly consistent with the original web page body area after rendering.

[0059] 2. Improve text extraction accuracy:

[0060] like Figure 4 As shown in the figure, this solution improves the text extraction accuracy by combining rule-based extraction and statistical feature extraction. By counting the tags, classes, and ids in the HTML text area of ​​the top 1000 popular sites, a text rule table is constructed. Common text area rules include: class="article", id="content", class="post-text".

[0061] When an HTML is input, it goes through HTML tag standardization and noise clipping, and first matches based on the text rules. When the extraction result does not trigger list noise clipping, if the current result non-a tag text length is > 20 characters, the result is returned directly. If the current result non-a tag text length is <= 20 characters and the non-a tag text length in the p tag of the entire HTML is > 100 characters, then switch to the extraction method based on statistical features.

[0062] When the extraction result triggers list noise clipping, if the current result non-a tag text length is > 50 characters, the result is returned directly; if the current result non-a tag text length is <= 50 characters and the non-a tag text length in the p tag of the entire HTML is > 100 characters, the statistical feature-based extraction method is switched.

[0063] Among them, the statistical feature-based extraction method first divides the content into blocks, and identifies the content blocks that may be the main text by analyzing the attributes of the nodes (such as tag name, class name, ID) and the relationship between the upper and lower nodes. Then the content blocks are scored and a score is assigned to each content block. This score is based on a series of factors, such as the length of the text in the block, the density of links, the density of stop words, the appearance of specific tags (such as , <article>etc. are considered positive signals, while <nav> , <aside>Finally, the best content blocks are selected, and the content blocks with the highest scores are selected, which are most likely to contain the main content of the article.

[0064] 3. Enhance multi-language support capabilities:

[0065] Some threshold indicators in the extraction method designed in this solution include text content length judgment. If only the len() method is used, the results of languages ​​such as Chinese and English will be very different. Therefore, for most languages ​​based on Latin letters, in order to ensure the accuracy of the threshold conditions of the extraction results, a special method of counting the number of words by separating spaces will be used to achieve the same content length judgment conditions as Chinese. When the threshold judgment conditions are consistent, since the extraction method proposed in this solution directly identifies the text area after converting HTML to DOM tree, it is not affected by the language type and has strong compatibility and versatility for multilingual content.

[0066] 4. Add the noise area cropping function based on HTML structure:

[0067] According to the HTML5 specification, some tags are not related to the text, such as <input> , <button>Since pure list pages generally do not contain content, this solution innovatively develops a method for identifying list areas (such as Figure 5 As shown). It contains multiple entries with a common parent node, and multiple entries form a list cluster. Common tags that can construct list clusters include li, div, ul, section, etc. When a common parent node contains multiple parallel child nodes and each child node contains a link a tag, when the number of child nodes meets >=5, the labels and attribute names of the child nodes, the labels and attribute names in the grandchild nodes are concatenated into string features, and the Jaccard similarity of the string features of each pair of child nodes is calculated. The lowest value is removed, and the average value of the remaining values ​​is the possibility that the current area is a list area. When this value is <0.85, it is regarded as a list area and removed.

[0068] 5. Optimize the noise area cropping based on style features:

[0069] This solution generates a noise area tag, class, and id keyword table by statistically analyzing the top 1000 popular sites. Common noise area style features include: style="display:none", class="advertisement", id="paywall". When an HTML tag attribute hits the above keyword table, it is scored based on the length of the text in the current tag, link density, and stop word density. When the score is lower than the preset score threshold, the current area is determined to be a noise area and is removed.

[0070] 6. LaTeX format conversion of web page formulas:

[0071] The LaTeX formulas rendered on HTML pages may become difficult to extract the original LaTeX code directly from the HTML source code due to different tools (such as MathJax, KaTeX, etc.) and their respective implementation methods. This is because these tools may use different methods to package LaTeX code, or perform some transformations on the original code during the rendering process. This extraction method designs statistics and summarizes the different methods of displaying latex formulas on a large number of formula-containing sites, and implements a conversion module with better compatibility.

[0072] Contains the following conversion methods:

[0073] 1. When the alt attribute of the image tag contains latex, insert a span node before the img tag node, and assign the text content to the img alt attribute content.

[0074] 2. When the type of the script tag is "math / tex" or "math / asciimath", convert the script tag to a span tag and assign the text content to the text content of the script.

[0075] 3. When the tag is math, build a local tag mapping table, convert the formula written in XML to standard latex, convert the math tag to span tag, and assign the text content to the converted standard latex.

[0076] 4. When the tag is mathjax, the asciimath2tex function is implemented to convert it to standard latex, the mathjax tag is converted to a span tag, and the text content is assigned to the converted standard latex.

[0077] 7. Improve extraction speed:

[0078] This solution reconstructs the processing flow, removes unnecessary steps, and no longer processes and generates results one by one based on a single HTML node, while avoiding the problem of multimodal image loss. The operation of cutting noise nodes is advanced, which greatly reduces the number of nodes judged and processed by the subsequent calculation and scoring of text area recognition, and reduces the complexity of the algorithm. The improved process is simpler and more efficient. The shortest path is to directly apply the rule, and the longest path is only from the rule to the statistical feature extraction, which greatly improves the processing speed. The processing library uses lxml to convert html to dom tree operations, and the performance is 10 times better than using BeautifulSoup.

[0079] To verify the effectiveness of this solution, this embodiment applies the above solution to the Intern WanJuan 1.0 and WanJuan 2.0 datasets. The data of the WanJuan series of image and text datasets mainly come from public web pages and CommonCrawl. The total number of documents processed by the web page content extraction method based on a combination of rules and statistical features exceeds 22 million, and relatively complete and high-quality image and text content can be obtained.

[0080] Intern WanJuan 1.0 has been applied to the training of large-scale models such as Intern Multimodal and Intern Puyu. By "digesting" high-quality corpus, the Intern series models have shown excellent performance in various generative tasks such as semantic understanding, knowledge question answering, visual understanding, and visual question answering. By comparing the 1B model training with the same type of datasets, using the perplexity of the validation dataset and the accuracy of downstream tasks as evaluation indicators, experiments have shown (as shown in Tables 1 and 2, where Table 1 is the perplexity comparison of different models on different validation sets, and Table 2 is the accuracy comparison of different models on different downstream tasks) that WanJuan 2.0 has significantly improved the performance of English text completion and general English proficiency tasks.

[0081] In addition, as shown in Table 3, the areas under the API perspective toxicity index curves of different data sets indicate that the data processed using this solution are safer.

[0082] Table 1

[0083] Area Under Curve of Toxicity Metrics by Perspective API for DifferentDatasets

[0084]

[0085] Table 2

[0086] Perplexity of Different Models on Diferent Validation Sets

[0087]

[0088] Table 3

[0089] Accuracy of Different Models on Different Downstream Tasks

[0090]

[0091] In summary, this solution significantly improves the quality and efficiency of web page content extraction through innovative methods. The following are the key features of this technical solution and its implementation mechanism:

[0092] 1. Keep the main area web page source code:

[0093] Implementation method: Use advanced parsing technology to completely retain the original format and information of elements such as pictures, paragraphs, code blocks, etc. in the web page, automatically identify and retain key content in the web page, such as pictures, paragraphs, and code blocks, etc., to ensure that the extracted content remains highly consistent with the original web page.

[0094] 2. Improve text extraction accuracy:

[0095] Implementation method: Optimize the algorithm to cope with the impact of multimedia elements such as pictures on text extraction accuracy. Through intelligent analysis and processing, more accurate text content extraction can be achieved. For complex layouts and multimedia elements in web pages, the accuracy and completeness of extraction can be improved.

[0096] 3. Enhance multi-language support capabilities:

[0097] Implementation method: Use advanced natural language processing technology to ensure that the system can provide consistent and efficient extraction results in multiple language environments (such as Chinese, English, Japanese, Arabic, etc.), especially for non-Latin languages.

[0098] 4. Added noise area cropping function based on HTML structure:

[0099] Implementation method: Identify and remove non-main content areas in web pages, such as list clusters, navigation bars, etc., to reduce noise interference and improve content quality.

[0100] 5. Optimize the noise area cropping based on style features:

[0101] Implementation method: Identify and remove non-essential content such as advertisements, comments, JavaScript and CSS from web pages to further improve the purity of the extracted content.

[0102] 6. LaTeX format conversion of web page formulas:

[0103] Implementation method: Integrate a dedicated mathematical formula recognition and conversion module to convert mathematical formulas in web pages into standard LaTeX format to facilitate subsequent data processing and analysis.

[0104] 7. Increase extraction speed:

[0105] Implementation method: Reconstruct the processing flow and remove unnecessary steps. For example, the result is no longer generated in units of HTML nodes, which avoids the loss of multimodal images. The improved process is more concise and efficient. The shortest path is to directly apply the rule, and the longest path is only from the rule to the statistical feature extraction, which greatly improves the processing speed and realizes the rapid extraction of web page content, ensuring the processing speed without losing the integrity of the content.

[0106] Through the above technical means, this solution not only significantly enhances the accuracy and comprehensiveness of the extracted content, in addition to effectively extracting regular web pages, it also supports the extraction of information and forum web pages, and also greatly improves the processing speed, which can provide users with a more efficient and accurate web content extraction tool.< / button> < / aside> < / nav> < / article> < / aside> < / nav> < / article>

Claims

1. A web page content extraction method based on a combination of rules and statistical features, characterized in that: The following steps are involved: S1. Convert the HTML source code of the web page into a DOM tree, retain the tree nodes identified as the text area, cut the tree nodes identified as the noise area, and save a copy of the DOM tree with the noise area removed; S2. Extract content from the saved DOM tree copy based on the combined switching method of rule extraction and statistical feature extraction.

2. A webpage content extraction method based on a combination of rules and statistical features according to claim 1, characterized in that: The step S1 comprises the following steps: S11, converting the web page HTML source code into a DOM tree, and then performing tag formatting and mathematical formula conversion processing; S12, for the DOM tree after the tag formatting and mathematical formula conversion processing, performing noise area clipping based on HTML structure and noise area clipping based on style features respectively; S13, saving a copy of the DOM tree with the noise area removed after cropping.

3. A webpage content extraction method based on a combination of rules and statistical features according to claim 2, characterized in that: The process of label formatting and mathematical formula conversion processing in step S11 includes: When the alt attribute of the image tag contains latex, a span node is inserted before the img tag node, and the text content is assigned to the img alt attribute content; When the type of the script tag is "math / tex" or "math / asciimath", convert the script tag to a span tag, and assign the text content to the text content of the script; When the tag is math, a local tag mapping table is constructed to convert the formula written in XML to standard latex, convert the math tag to a span tag, and assign the text content to the converted standard latex. When the tag is mathjax, the asciimath2tex function is implemented to convert it into standard latex, convert the mathjax tag into a span tag, and assign the text content to the converted standard latex.

4. A webpage content extraction method based on a combination of rules and statistical features according to claim 2, characterized in that: The HTML structure-based noise region cropping in step S12 is specifically to automatically identify and remove non-main content regions in the web page, including list clusters and navigation bars, by calculating the HTML tag set and text tag density.

5. A webpage content extraction method based on a combination of rules and statistical features according to claim 4, characterized in that: The process of cutting out the noise area based on the HTML structure in step S12 includes: For entries with multiple common parent nodes, where multiple entries constitute a list cluster, the labels for constructing the list cluster include li, div, ul, and section. When a common parent node contains multiple parallel child nodes and each child node contains a link a label, when the number of child nodes satisfies>=5, the labels and attribute names of the child nodes, the labels and attribute names in the grandchild nodes are concatenated into string features, and the Jaccard similarity of the string features of each pair of child nodes is calculated. The lowest value is removed, and the average of the remaining values ​​is the possibility that the current area is a list area. When this value is <0.85, it is regarded as a list area and is removed.

6. A webpage content extraction method based on a combination of rules and statistical features according to claim 2, characterized in that: The noise region cropping based on style features in step S12 specifically generates noise region labels, class, and id keyword tables through statistical analysis, and identifies and removes non-essential content in the web page, including advertisements, comments, JavaScript, and CSS.

7. A webpage content extraction method based on a combination of rules and statistical features according to claim 6, characterized in that: The process of cutting out the noise area based on the style feature in step S12 includes: Statistical analysis of the top 1000 popular sites is performed to generate a noise area tag, class, and id keyword table. The noise area style features include: style="display:none", class="advertisement", id="paywall". When an HTML tag attribute hits the above keyword table, it is scored based on the length of the text in the current tag, link density, and stop word density. When the score is lower than the preset score threshold, the current area is determined to be a noise area and is removed.

8. The webpage content extraction method based on the combination of rules and statistical features according to claim 1, characterized in that: The combined switching process based on rule extraction and statistical feature extraction in step S2 includes: Count the tags, classes, and ids in the HTML text area of ​​the top 1000 popular sites, and build a text rule table. The text area rules include: class="article", id="content", class="post-text"; When an HTML is input, after HTML tag standardization and noise clipping, it is first matched based on the text rules. When the extraction result does not trigger the list noise clipping, if the current result non-a tag text length is > 20 characters, the extraction result of the web page body area content is directly returned; if the current result non-a tag text length is <= 20 characters and the non-a tag text length in the p tag of the entire HTML is > 100 characters, then switch to the extraction method based on statistical features to extract the web page body area content; When the extraction result triggers list noise trimming, if the current result non-a tag text length is > 50 characters, the extraction result of the web page body area content is directly returned; if the current result non-a tag text length is <= 50 characters and the non-a tag text length in the p tag in the entire HTML is > 100 characters, then switch to the statistical feature-based extraction method to extract the web page body area content.

9. A webpage content extraction method based on a combination of rules and statistical features according to claim 8, characterized in that: The specific process of extracting the content of the web page body area by the statistical feature-based extraction method is as follows: First, divide the content into blocks: analyze the properties of the nodes and the relationship between the upper and lower nodes, and identify the content blocks containing the main text; Then, score the content blocks: assign a corresponding score to each content block; Finally, select the best content block: select the content block with the highest score, which contains the main content of the article.

10. A webpage content extraction method based on a combination of rules and statistical features according to claim 9, characterized in that: The content block score is specifically calculated based on the length of the text in the block, the link density, the stop word density, and the appearance of specific tags. , <article>, it is considered a positive signal and is given extra points. <nav> , <aside> , then points will be deducted.< / aside> < / nav> < / article>

Images