Label score-based webpage information automatic extraction method and device

Abstract

The invention discloses an automatic webpage information extraction method based on label scores. Deleting an HTML (Hypertext Markup Language) file of the webpage, gt; and non-target content of the tag. And deleting the HTML element where the HTML tag of which the attribute name comprises the non-core text keyword is located in the remaining HTML file of the webpage. All the lt is extracted from the remaining HTML files of the webpage; gt; a label, and according to lt; gt; and constructing an XPath statement according to a nesting relationship between the tag and a parent tag of the tag. All the XPath statements are scored, and one or more XPath statements with the highest scoring result are reserved. In the reserved XPath statements, the result is detected; gt; the attribute value and the result of the href attribute in the tag; gt; calculating the text similarity of the labeled element content; and filtering the XPath statement. The webpage analysis method and device have high webpage analysis accuracy on the premise of ensuring that data missing does not occur.

Description

Technical Field

[0001] The present application relates to a method for extracting web page information. Background Art

[0002] A website is a collection of interrelated (usually interlinked) web pages, which are usually in HTML (HyperText Markup Language) format. Different websites have different HTML structures.

[0003] When using a crawler program to automatically crawl text on a web page, the crawler program will parse all the hypertext links (referred to as hyperlinks) on the web page and access these hyperlinks according to a set of strategies. If these hyperlinks point to articles, the crawler program automatically crawls and saves the article information.

[0004] Currently, there are three main methods for automatic extraction of web page information, each with its own shortcomings.

[0005] The first is manual configuration, where a web page parsing rule specifically adapted to each website is created manually. For example, an XPath (XML Path Language) is used to configure a rule in the form of “parent tag [attribute name] / / child tag [attribute name] / Tags" to locate the extraction. XPath is used to find information in XML (eXtensible Markup Language) documents and is also applicable to HTML documents. If the number of websites to be parsed is large, it will take a lot of manpower to write different web page parsing rules based on the HTML structure of each website.

[0006] The second is rule matching, which means building some fixed parsing templates in advance. For example, many websites' a tags are under the li tag under the div tag, or under the li tag under the ul tag, or under the td tag under the table tag. Then, multiple fixed parsing templates are built based on these common HTML structures, and these parsing templates are used when parsing the website. If the HTML structure of the website meets the rules of the preset parsing template, the parsing will be successful. If the HTML structure of the website does not meet the rules of the preset parsing template, the parsing will fail. Therefore, this technology has problems such as low accuracy and poor versatility.

[0007] The third method is to use a natural language processing (NLP) model. Model training relies on a large amount of data annotation to form a training data set, and running an NLP model also requires high hardware resources such as graphics cards. If a trained large language model (LLM) is used, although data annotation is not required, the cost of calling a trained large model is very high due to the huge amount of web information resources.

[0008] In general, existing automatic web page information extraction technologies are difficult to strike a balance between cost and accuracy.

[0009] In addition, the hyperlinks in web pages do not always point to articles with information extraction value. Some hyperlinks point to multimedia content, such as pictures, animations, audio, video, etc. Some hyperlinks point to navigation bars, headers, footers, etc. Therefore, many hyperlinks have no information extraction value, but occupy a lot of time and processing resources of the crawler program, reducing the information extraction efficiency of the crawler program. Summary of the invention

[0010] The technical problem to be solved by this application is: how to realize a universal method for automatically extracting web page information that can adapt to different websites, and can take into account both cost and accuracy to improve information extraction efficiency.

[0011] To solve the above technical problems, the present application proposes a method for automatically extracting web page information based on tag scoring, which includes the following steps: Step S1: Delete the HTML file of the web page that cannot contain Step S2: Delete the HTML element containing the HTML tag whose attribute name contains the non-core text keyword in the remaining HTML file of the web page. Step S3: Extract all the Tags, and according to The nested relationship between a tag and its parent tag constructs an XPath statement; The parent tag of a tag is Step S4: Score all XPath statements and retain the one or more XPath statements with the highest scores. Step S5: Score the XPath statements that are retained. The attribute value of the href attribute in the tag is The text similarity of the element content after the tag is calculated; if the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted; if the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

[0012] Furthermore, in step S1, the non-target content includes empty elements and other HTML elements that cannot embed a elements.

[0013] Exemplarily, in step S2, the non-core text keywords include any one of bottom, footer, hover, link, login, logo, and top.

[0014] Furthermore, in step S3, all the remaining HTML files of the web page are first extracted. Tags, and then extract these All parent HTML tags of the tag in the remaining HMTL files, and then according to all the parent HTML tags of the remaining HMTL files The nested relationship between the tag and its parent HTML tag constructs all the XPath statements that actually exist in the remaining HMTL file.

[0015] Furthermore, in step S3, the number of XPath statements constructed depends on two factors: first, the number of XPath statements in the remaining HTML files. How many parent HTML tags are there for a tag? Second, how many parent HTML tags are there for a tag? What is the actual nesting relationship between the tag and its parent HTML tag.

[0016] Preferably, in step S4, firstly The attribute value of the href attribute in the tag is parsed. If the parsing result is empty, the XPath statement is deleted; the remaining XPath statements whose parsing results are not empty are scored.

[0017] Furthermore, in step S4, the scoring of the XPath statement is based on the following scoring rules. First, if If the attribute value of the href attribute in the tag points to a file, multimedia content, the homepage of a website, or a list link, the XPath statement is judged as negative. The attribute value of the href attribute in the tag points to the link of the article page, then the XPath statement is judged as positive. If the attribute value of the href attribute in the tag points to a link to a non-article page, the XPath statement is considered negative. The attribute name of the parent tag of the tag points to the article, then the XPath statement is judged as positive. Fifth, if the XPath statement If the element content after the tag points to a non-article, the XPath statement is judged as negative. If the XPath statement is judged as positive once, one point is added; if it is judged as positive multiple times, multiple points are added; if it is judged as negative, no points are added or subtracted.

[0018] Exemplarily, in step S4, the first rule includes: The attribute value of the href attribute in the tag contains any of the suffixes ".pdf", ".js", ".jpg", ".zip", ".doc", ".gif", ".com", ".cn", or ends with a forward slash " / ". The second rule includes: The attribute value of the href attribute in the tag contains any of the suffixes ".html", ".htm", ".php", ".asp", ".jsp", ".aspx", ".shtml", ".xml", and ".jhtml". The third rule includes: The attribute value of the href attribute in the tag contains any of the text "javascript", "void", "login", "register", "url=http", "url=map", "url=password". The fourth rule includes: The attribute name of the parent tag of the tag contains any of the text "list", "news", "lst", "content", "box", "cont". The fifth rule includes: The element content after the tag includes any of the texts "Home", "Portal", "Copyright", "Map", "Email", "Register", "Login", "Accessibility", "Website Statement", "About Us", "Last Page", "Next Page", "Government Services", "Public Services", "Contact Us", "Jump", "Last Page", "Previous Page", "Previous Page", "Next Page", "Download Center", "Network Security", "Address", "Opinion Collection", "Telephone", "Postal Code", "Postal Code", "Links", "Public Network Security", "Last Page", "Mobile Version", and "Website Map".

[0019] Furthermore, if the target content cannot be obtained after step S5 is executed, the HTML element containing the HTML tag whose attribute name contains the non-core text keyword deleted in step S2 is restored, and steps S3 to S5 are repeated.

[0020] The present application also proposes a web page information automatic extraction device based on tag scoring, including a non-target content deletion unit, a low-value content deletion unit, an XPath statement construction unit, an XPath statement scoring filtering unit, and a text similarity filtering unit. The non-target content deletion unit is used to delete the content that is impossible to contain in the HTML file of the web page. The low-value content deletion unit is used to delete the HTML element where the HTML tag whose attribute name contains the non-core text keyword is located in the remaining HTML file of the web page. The XPath statement construction unit is used to extract all the tags, and based on these The nested relationship between a tag and its parent tag constructs an XPath statement; The parent tag of a tag is The XPath statement scoring filter unit is used to score the XPath statements and retain the one or more XPath statements with the highest scoring results. The text similarity filter unit is used to filter the XPath statements retained. The attribute value of the href attribute in the tag is The text similarity of the element content after the tag is calculated; if the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted; if the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

[0021] The technical effect achieved by the present application is that the traditional method of manually configuring XPath statements is completely dependent on manual labor, while the present application has a higher accuracy of web page parsing while ensuring that no data is missed, and is completely independent of manual labor, thus greatly reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of an HMTL element.

[0023] Figure 2 This is a diagram of another HMTL element (empty element).

[0024] Figure 3It is a schematic diagram of the nesting between HMTL elements.

[0025] Figure 4 It is a flowchart of the method for automatic extraction of web page information based on tag scoring proposed in this application.

[0026] Figure 5 is an example of an original web page.

[0027] Figure 6 Yes Figure 5 The HTML file of the web page shown is an example of a web page corresponding to the remaining HTML file obtained after the HTML file of the web page executes step S1 and step S2.

[0028] Figure 7 It is a structural diagram of the device for automatically extracting web page information based on tag scoring proposed in this application.

[0029] Explanation of the reference numerals in the figure: non-target content deletion unit 1, low-value content deletion unit 2, XPath statement construction unit 3, XPath statement scoring filtering unit 4, text similarity filtering unit 5. DETAILED DESCRIPTION

[0030] The HTML structure and content of a web page are defined by HTML files, and HTML files are defined by HTML elements.

[0031] See also Figure 1 This is an HMTL element, which consists of a start tag, element content, and an end tag. The start tag includes the element name (p in this case, so this is a p element) and element attributes (optional, not in this case), as well as a pair of angle brackets on the outermost edge. The end tag is similar to the start tag, except that a forward slash " / " is added before the element name. The element content is between the start tag and the end tag. The start tag and the end tag are collectively referred to as HMTL tags. Some HTML elements have no element content, which is called empty content.

[0032] See also Figure 2 , which is another HTML element, called a void element. A void element has only a start tag, no element content, and no end tag. HTML tags in a void element always appear alone, for example Tags. HMTL tags in HTML elements other than empty elements always appear in pairs, such as Tags and Label.

[0033] Most HTML elements can have attributes, which contain extra information about the element that you don't want to appear in the element's content. Figure 2The HTML element shown has two attributes. Each attribute usually consists of an attribute name before the equal sign and an attribute value in double quotes after the equal sign. Some attributes do not have an equal sign and an attribute value. Because the attributes of HTML elements are located in the angle brackets of HTML tags, the attributes of HTML elements are sometimes called attributes of HTML tags. For example, The tag has an href attribute, which is used to create a hypertext link. The attribute value of the href attribute is a URL (Uniform Resource Locator) address.

[0034] Most HTML elements can be nested. See Figure 3 , which is an HTML element that contains Figure 1 Another HTML element as shown. However, empty elements cannot contain other HTML elements.

[0035] See also Figure 4 ,The method for automatic extraction of web page information based on label scoring proposed in this application includes the following steps.

[0036] Step S1: Delete the content that cannot be included in the HTML file of the web page. Non-target content of the tag. First, other HTML elements cannot be embedded in empty elements, that is, a single HTML tag cannot contain Tags are non-target content. These empty elements include input elements, link elements, meta elements, etc. Secondly, according to the HTML syntax structure, some HTML elements cannot embed a elements, that is, some paired HMTL tags cannot exist between them. Tags, these paired HMTL tags and the element content in between are also non-target content. These paired tags cannot contain The HTML tags of the tag include, for example, the tag and the tag, <script>标签和< / script> Tags, etc.

[0037] Step S2: Determine whether the attribute names of all HTML tags in the remaining HTML files of the web page contain keywords such as bottom, footer, hover, link, login, logo, top, etc. The HTML tags containing the attribute names of these keywords belong to the non-core text of the web page. Tags are not very valuable for information extraction, so the HTML elements containing HTML tags whose attribute names contain non-core text keywords are deleted.

[0038] For example, the opening tag of an HTML element is <div class＝”footer”id＝”footer”> , followed by the element content, followed by the closing tag Because the opening tag of the HTML element contains the keyword footer, the HTML element is deleted.

[0039] Steps S1 and S2 are used to filter invalid text and remove redundant information from the HTML file of the web page. The difference between these two steps is that step S1 removes the content that is definitely not included in the HTML file. The non-target content of the tag, step S2 deletes the content that may contain Content that is not valuable for information extraction. Figure 5 , which is the original page. See Figure 6 , this is right Figure 5 The HTML file of the web page shown executes step S1 and step S2 to obtain the web page corresponding to the remaining HTML file. Figure 5 The title bar at the top of the web page, the navigation bar on the left side of the web page, and the small icons on the right side of the web page are all removed. Figure 6 What is retained are hypertext links with greater value for information extraction.

[0040] Step S3: Extract all the remaining HTML files in the web page Tags, and based on these in the remaining HMTL files The nested relationship between a tag and its parent tag constructs an XPath statement. The parent tag of a tag is One or more levels of HTML tags above the tag. The first level of HTML tags above the tag directly contains The HTML tag of the tag is called HTML tag one; The second-level HTML tag above the tag is the HTML tag that directly contains HTML tag one, and so on.

[0041] For example, this step first extracts all the remaining HTML files on the web page. Tags, and then extract these All parent HTML tags of the tag in the remaining HMTL files, assuming that the There are five parent HTML tags for tags, namely Label, Label, <h1>Label, Label, Tags. Then, based on all the remaining HTML files The nested relationship between the tag and its parent HTML tag constructs all XPath statements that actually exist in the remaining HTML file. These XPath statements are in the form of " / / div / / div / a", for example, which means that a The first level of HTML tags above the tag is Tags, the One or more levels of HTML tags above the tag or Tags. A single slash " / " indicates a single-layer nesting relationship, and a double slash " / / " indicates a single-layer or multi-layer nesting relationship. In this example, there are up to 25 possible XPath statements, namely " / / div / / div / a", " / / div / / li / a", " / / div / / h1 / a", " / / div / / table / a", " / / div / / td / a", " / / li / / div / a", " / / li / / li / a", " / / li / / h1 / a", " / / li / / table / a", " / / li / / td / a", " / / h1 / / div / a", " / / h1 / / li / a "、" / / h1 / / h1 / a"," / / h1 / / table / a"," / / h1 / / td / a"," / / table / / div / a"," / / table / / li / a"," / / table / / h1 / a"," / / table / / table / a"," / / table / / td / a"," / / td / / div / a"," / / td / / li / a"," / / td / / h1 / a"," / / td / / table / a"," / / td / / td / a". In other application scenarios, the number of XPath statements constructed in this step depends on two factors. The first is the number of XPath statements in the remaining HMTL files. How many parent HTML tags are there for a tag? Second, how many parent HTML tags are there for a tag? What is the actual nesting relationship between the tag and its parent HTML tag.

[0042] Step S4: Score all XPath statements. The attribute value of the href attribute in the tag (which is a URL address) is parsed. If the parsing result is empty, the XPath statement is deleted; the remaining XPath statements whose parsing results are not empty are scored. The scoring of the XPath statement in this step is the sum of the following scoring rules established according to the construction rules and habits of HTML.

[0043] First, the XPath statements involved in scoring If the attribute value of the href attribute in the tag contains any of the suffixes ".pdf", ".js", ".jpg", ".zip", ".doc", ".gif", ".com", ".cn", or ends with a forward slash " / ", it is judged as negative. These suffixes either point to files, multimedia content, or website homepages or list links (when the attribute value of the href attribute ends with a forward slash " / "), and no useful information can be extracted, or the information extracted is of little value.

[0044] Second, the XPath statements involved in scoring If the attribute value of the href attribute in the tag contains any of the suffixes ".html", ".htm", ".php", ".asp", ".jsp", ".aspx", ".shtml", ".xml", and ".jhtml", it is judged as positive. These suffixes usually point to articles that may have information extraction value.

[0045] Third, the XPath statements involved in scoring If the attribute value of the href attribute in the tag contains any of the texts "javascript", "void", "login", "register", "url=http", "url=map", "url=password", it is considered negative. These texts usually point to links to pages related to redirection, login, registration, maps, etc., and have no information extraction value.

[0046] Fourth, the XPath statements involved in scoring If the attribute name of the parent tag of a tag contains any of the texts "list", "news", "lst", "content", "box", and "cont", it is considered positive. These texts usually represent the core content and point to articles with information extraction value. The parent tag of a tag is One or more levels of HTML tags above the tag.

[0047] Fifth, the XPath statements involved in scoring If the element content after the tag (i.e. the element content of the a element, which is a paragraph of text) contains any of the texts "Home", "Portal", "Copyright", "Map", "Email", "Registration", "Login", "Accessibility", "Website Statement", "About Us", "Last Page", "Next Page", "Government Service", "Public Service", "Contact Us", "Jump", "Last Page", "Previous Page", "Previous Page", "Next Page", "Download Center", "Network Security", "Address", "Opinion Collection", "Telephone", "Postal Code", "Postal Code", "Friendly Links", "Public Network Security", "Last Page", "Mobile Version", and "Website Map", these texts usually indicate that they are not detailed articles, and are therefore judged as negative. The directions of these texts usually have no information extraction value.

[0048] The XPath statements involved in the scoring are scored by the above multiple rules at the same time. If they are judged as positive once, they will be scored once. If they are judged as positive multiple times, they will be scored multiple times. If they are judged as negative, they will neither be scored nor deducted. Finally, one or more XPath statements with the highest scoring results are retained.

[0049] Step S5: In the retained XPath statement The attribute value of the href attribute in the tag is The element content after the tag calculates text similarity. If the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted. If the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

[0050] The significance of this step of text similarity comparison is to filter out the column links in the navigation bar in the HTML file. The column links in the navigation bar are very similar to the article links in terms of HTML tags, attributes, and text. It is difficult for each scoring rule (screening logic) in step S4 to filter out these links, which is a difficult point to judge. Only when there is a relatively obvious difference in text similarity can they be successfully filtered out.

[0051] Subsequent crawlers extract If we extract information from the attribute value of the href attribute in the tag, we have a greater chance of extracting valuable text information. When you use the tag, only the content that is "100% not the target content" will be filtered out. Tags, retaining any possible target content Labels sacrifice some accuracy to ensure that no data is missed, so the threshold should be set low.

[0052] If step S5 ultimately fails to obtain the target content, the HTML element containing the HTML tag with the attribute name containing the non-core text keyword deleted in step S2 is restored, and steps S3 to S5 are repeated.

[0053] Figure 4 In the method shown, steps S1 to S2 belong to the text preprocessing stage, and steps S3 to S5 belong to the information extraction stage. In the text preprocessing stage, the present application designs a fine-grained filtering rule to remove irrelevant information from the web page content. In the information extraction stage, the present application designs an information extraction method based on HTML tag scoring. First, all the Tags and their parent tags and construct XPath statements, and then score the XPath statements based on scoring rules. This application conducts word frequency statistics and experience summarization on tens of thousands of Chinese web pages, and designs scoring rules based on the judgment of tags, attributes, text and other information in HTML files. It can accurately filter out non-target content such as navigation bars, pictures, headers, and footers in web page content. The XPath statement with the highest score is the target content to be extracted after text similarity comparison.

[0054] See also Figure 7 The web page information automatic extraction device based on tag scoring proposed in this application includes a non-target content deletion unit 1, a low-value content deletion unit 2, an XPath statement construction unit 3, an XPath statement scoring filtering unit 4, and a text similarity filtering unit 5. Figure 7 The device shown corresponds to Figure 4 The method shown.

[0055] The non-target content deletion unit 1 is used to delete the non-target content that cannot be included in the HTML file of the web page. Non-target content of a tag.

[0056] The low-value content deletion unit 2 is used to delete the HTML element where the HTML tag whose attribute name contains the non-core text keyword is located in the remaining HTML file of the web page.

[0057] The XPath statement building unit 3 is used to extract all the remaining HTML files of the web page. tags, and based on these The nested relationship between a tag and its parent tag builds the XPath statement.

[0058] The XPath statement scoring and filtering unit 4 is used to score the XPath statements and retain one or more XPath statements with the highest scoring results.

[0059] The text similarity filtering unit 5 is used to filter the text similarity in the retained XPath statement. The attribute value of the href attribute in the tag is The element content after the tag calculates text similarity. If the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted. If the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

[0060] It should be noted that although the present application also constructs XPath statements, it does not belong to the existing manually configured automatic extraction methods of web page information. The existing manually configured automatic extraction methods of web page information need to analyze the HTML layout of each website and write different web page parsing rules for each website. The present application uses the universal logic of "parsing all data" combined with "scoring filtering" combined with "text similarity filtering" to complete the parsing of all websites. Therefore, the present application is completely free from manual labor, which greatly reduces labor costs. In addition, the present application uses two methods, scoring and text similarity comparison, to filter XPath statements, which can greatly improve the accuracy of web page parsing; a lower threshold is used when comparing text similarity to ensure that no data is missed.

[0061] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application. < / h1>

Claims

1. A method for automatically extracting web page information based on tag scoring, characterized in that: The method comprises the following steps: Step S1: Delete the content that cannot be included in the HTML file of the web page. Non-target content of tags; Step S2: deleting the HTML element containing the HTML tag whose attribute name contains the non-core text keyword from the remaining HTML files of the web page; Step S3: Extract all the remaining HTML files in the web page Tags, and according to The nested relationship between a tag and its parent tag constructs an XPath statement; The parent tag of a tag is One or more levels of HTML tags above the tag; Step S4: Score all XPath statements, and retain one or more XPath statements with the highest score results; Step S5: In the retained XPath statement The attribute value of the href attribute in the tag is The text similarity of the element content after the tag is calculated; if the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted; if the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

2. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: In step S1, the non-target content includes empty elements and other HTML elements that cannot embed a elements.

3. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: In step S2, the non-core text keywords include any one of bottom, footer, hover, link, login, logo, and top.

4. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: In step S3, first extract all the remaining HTML files of the web page. Tags, and then extract these All parent HTML tags of the tag in the remaining HMTL files, and then according to all the parent HTML tags of the remaining HMTL files The nested relationship between the tag and its parent HTML tag constructs all the XPath statements that actually exist in the remaining HMTL file.

5. The method for automatically extracting web page information based on tag scoring according to claim 4 is characterized in that: In step S3, the number of XPath statements constructed depends on two factors: first, the number of XPath statements in the remaining HTML files. How many parent HTML tags are there for a tag? Second, how many parent HTML tags are there for a tag? What is the actual nesting relationship between the tag and its parent HTML tag.

6. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: In step S4, first, all XPath statements in The attribute value of the href attribute in the tag is parsed. If the parsing result is empty, the XPath statement is deleted; the remaining XPath statements whose parsing results are not empty are scored.

7. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: In step S4, the scoring of the XPath statement is based on the following scoring rules: First, if the XPath statement If the attribute value of the href attribute in the tag points to a file, or to multimedia content, or to the homepage of a website or a list link, the XPath statement is judged as negative; Second, if the XPath statement If the attribute value of the href attribute in the tag points to a link to the article page, the XPath statement is considered positive; Third, if the XPath statement If the attribute value of the href attribute in the tag points to a link to a non-article page, the XPath statement is considered negative; Fourth, if the XPath statement If the attribute name of the parent tag of the tag points to the article, the XPath statement is judged as positive; Fifth, if the XPath statement If the element content after the tag points to a non-article, the XPath statement is considered negative; If the Xpath statement is judged as positive once, one point will be added; if it is judged as positive multiple times, multiple points will be added; if it is judged as negative, no points will be added or subtracted.

8. The method for automatically extracting web page information based on tag scoring according to claim 7 is characterized in that: In step S4, the first rule includes: The attribute value of the href attribute in the tag contains any of the suffixes ".pdf", ".js", ".jpg", ".zip", ".doc", ".gif", ".com", ".cn", or ends with a forward slash " / "; The second rule includes: The attribute value of the href attribute in the tag contains any of the suffixes ".html", ".htm", ".php", ".asp", ".jsp", ".aspx", ".shtml", ".xml", and ".jhtml"; The third rule includes: The attribute value of the href attribute in the tag contains any one of the texts "javascript", "void", "login", "register", "url=http", "url=map", "url=password"; The fourth rule includes: The attribute name of the parent tag of the tag contains any of the texts "list", "news", "lst", "content", "box", and "cont"; The fifth rule includes: The element content after the tag includes any of the texts "Home", "Portal", "Copyright", "Map", "Email", "Register", "Login", "Accessibility", "Website Statement", "About Us", "Last Page", "Next Page", "Government Services", "Public Services", "Contact Us", "Jump", "Last Page", "Previous Page", "Previous Page", "Next Page", "Download Center", "Network Security", "Address", "Opinion Collection", "Telephone", "Postal Code", "Postal Code", "Links", "Public Network Security", "Last Page", "Mobile Version", and "Website Map".

9. The method for automatically extracting web page information based on tag scoring according to claim 1 is characterized in that: If the target content cannot be obtained after step S5 is executed, the HTML element containing the HTML tag with the attribute name containing the non-core text keyword deleted in step S2 is restored, and steps S3 to S5 are repeated.

10. A device for automatically extracting web page information based on tag scoring, characterized in that: It includes a non-target content deletion unit, a low-value content deletion unit, an XPath statement construction unit, an XPath statement scoring filtering unit, and a text similarity filtering unit; The non-target content deletion unit is used to delete the non-target content that cannot be included in the HTML file of the web page. Non-target content of tags; The low-value content deletion unit is used to delete the HTML element where the HTML tag whose attribute name contains the non-core text keyword is located in the remaining HTML file of the web page; The XPath statement building unit is used to extract all the remaining HTML files in the web page. tags, and based on these The nested relationship between a tag and its parent tag constructs an XPath statement; The parent tag of a tag is One or more levels of HTML tags above the tag; The XPath statement scoring and filtering unit is used to score the XPath statements and retain one or more XPath statements with the highest scoring results; The text similarity filtering unit is used to filter the text similarity in the retained XPath statement. The attribute value of the href attribute in the tag is The text similarity of the element content after the tag is calculated; if the text similarity calculated by a certain XPath statement is less than a certain threshold, the XPath statement is deleted; if the text similarity calculated by a certain XPath statement is greater than or equal to the threshold, the XPath statement is used as the final filtered target content.

Images