Rule identification method and device, electronic equipment, storage medium and program product
By automatically identifying XPath rules, combining a tag and text length and tag attribute characteristics, the problems of low manual recognition efficiency and insufficient accuracy are solved, and efficient data collection and rule recognition are achieved.
Patent Information
- Application Number
- CN202411188715.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-17
AI Technical Summary
In the prior art, the efficiency of manually identifying XPath rules is low and prone to human errors, resulting in low data collection efficiency and insufficient accuracy.
By obtaining the a tag of the target web page, determining the XPath list, and automatically generating the list XPath rules based on text length and preset keywords, combining the tag attribute characteristics and label density algorithm, automatically identifying page turn rules and details rules, realizing the automation of XPath rules Identify.
It improves the recognition efficiency of XPath rules, reduces manual operations, avoids human errors, and enhances the degree of automation of data collection and the adaptability of rules.
Smart Images
Figure CN119149800A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a rule recognition method, device, electronic device, storage medium and program product. Background Art
[0002] Faced with the massive amount of data on the Internet, using Internet crawler technology to obtain data has become an inevitable trend in big data analysis and data mining. The essence of web crawler technology is to simulate human browsing behavior on the Internet through automated programs to collect and extract information on specific websites. Among them, obtaining information on specific websites by configuring the XPath (XML PathLanguage) rules or regular expressions of web pages is a key technology. Therefore, how to quickly configure the XPath rules of seed websites is one of the research directions of automated data collection.
[0003] In related technologies, data collection on public source websites is mostly achieved by manually identifying and configuring XPath rules of web pages. The manually configured XPath rules are synchronized to the crawler server through the client for rule management. The crawler tool can read the XPath rules of the crawler server for data collection.
[0004] However, the method of manually identifying XPath rules is inefficient and prone to human errors. Summary of the invention
[0005] The embodiments of the present application provide a rule recognition method, device, electronic device, storage medium and program product to solve the technical problems that the existing manual XPath rule recognition method is inefficient and prone to human errors.
[0006] In a first aspect, an embodiment of the present application provides a rule identification method, including: obtaining a target web page; determining an XPath list based on the a tag of the target web page; determining a list XPath rule based on the text length of the target web page, preset keywords and the XPath list; determining a page turning XPath rule; obtaining web page information based on the list XPath rule; and determining a detail XPath rule based on the web page information.
[0007] In one embodiment, based on the a tag of the target web page, an XPath list is determined, including: based on the a tag of the target web page and the text where the a tag is located, recursively obtaining the parent element of the a tag; determining each child element of the parent element; determining the hierarchical structure of each child element; and determining the XPath list based on the parent element, each child element and the hierarchical structure of each child element.
[0008] In one embodiment, based on the text length of the target webpage, preset keywords and the XPath list, a list XPath rule is determined, including: based on the text length of the target webpage and the preset keywords, filtering out matching XPath rules from the XPath list to generate an XPath rule set; generalizing the XPath rule set to obtain a generalized XPath rule set; counting XPath clusters in the generalized XPath rule set to obtain a counting result of each XPath cluster in the generalized XPath rule set, and sequentially sorting the counting results of each XPath cluster, extracting the XPath cluster with the largest counting result as the optimal XPath rule; based on the optimal XPath rule, searching for matching XPath rules in the XPath list, and combining and generating a list rule based on the matching XPath rules in the XPath list; based on the optimal XPath rule, searching for a target XPath rule whose class attribute and / or ID attribute meets the conditions, and replacing and combining the list rules based on the target XPath rule to generate a list XPath rule.
[0009] In one embodiment, the page turning XPath rule includes a first page turning rule and a second page turning rule, the first page turning rule is a page turning rule based on a GET request, and the second page turning rule is a page turning rule based on a POST request; determining the page turning XPath rule includes: determining a tag to be identified that contains consecutive numbers on a target web page; judging whether the tag to be identified is an a tag; if the tag to be identified is an a tag, obtaining the URL of the tag to be identified, and determining the first page turning rule based on the URL of the tag to be identified; if the tag to be identified is not an a tag, determining the second page turning rule based on the tag attribute characteristics of the tag to be identified.
[0010] In one embodiment, the web page information includes a detail web page URL, a list title, and a publishing time; based on the list XPath rule, the web page information is obtained, including: based on the list XPath rule, the detail web page URL is obtained; the text of the a tag where the list XPath rule is located is used as the list title; all sibling elements of the a tag where the list XPath rule is located and the text of each sibling element are obtained; based on each sibling element and the text of each sibling element, the publishing time is obtained through a time regular expression.
[0011] In one embodiment, based on the web page information, the details XPath rule is determined, including: based on the details web page URL, downloading the details web page; based on the list title, determining the title element position in the details web page; based on the release time, determining the release time position in the details web page; based on the title element position and the release time position, obtaining the first text element in the details web page; based on the text tag attribute characteristics, determining the second text element in the details web page; based on the tag density algorithm, determining the third text element in the details web page; based on preset business keywords, determining the fourth text element in the details web page; based on the first text element, the second text element, the third text element and the fourth text element, determining the details XPath rule.
[0012] In a second aspect, an embodiment of the present application provides a rule identification device, including: an acquisition module, used to acquire a target web page; a list rule determination module, used to determine an XPath list based on the a tag of the target web page; based on the text length of the target web page, preset keywords and the XPath list, determine the list XPath rule; a page turning rule determination module, used to determine the page turning XPath rule; a detail rule determination module, used to acquire web page information based on the list XPath rule; based on the web page information, determine the detail XPath rule.
[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the rule recognition methods described above is implemented.
[0014] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned rule recognition methods.
[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned rule recognition methods when executed by a processor.
[0016] The rule recognition method, device, electronic device, storage medium and program product provided in the embodiments of the present application obtain the target web page; determine the XPath list based on the a tag of the target web page; determine the list XPath rule based on the text length of the target web page, preset keywords and the XPath list; determine the page turning XPath rule; obtain web page information based on the list XPath rule; determine the detail XPath rule based on the web page information. Through the above method, the automatic recognition of XPath rules can be realized without manual operation, which can improve the recognition efficiency of XPath rules and avoid human errors. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 This is one of the flow charts of the rule identification method provided in the embodiment of the present application.
[0019] Figure 2 This is the second flowchart of the rule identification method provided in the embodiment of the present application.
[0020] Figure 3 It is a flowchart of the identification list XPath rule provided in the embodiment of the present application.
[0021] Figure 4 It is a flowchart of identifying page turning XPath rules provided in an embodiment of the present application.
[0022] Figure 5 It is a flowchart of the identification detail XPath rule provided in the embodiment of the present application.
[0023] Figure 6 It is a structural diagram of a rule recognition device provided in an embodiment of the present application.
[0024] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0026] See also Figure 1 , Figure 1 This is one of the flow charts of the rule identification method provided in the embodiment of the present application. In the embodiment of the present application, the rule identification method is used for data collection of a public source website with a list page. The rule identification method includes steps S110 to S160, and each step is as follows: S110: Acquire a target web page.
[0027] See also Figure 2 , Figure 2This is the second flowchart of the rule identification method provided in the embodiment of the present application.
[0028] like Figure 2 As shown, the rule identification method of this embodiment can automatically identify the XPath rules of the web pages of the public information source website with list pages, wherein the XPath rules to be identified include list XPath rules, page turning XPath rules and detail XPath rules.
[0029] Optionally, the target web page's list XPath rule, detail web page URL (Uniform Resource Locator), list title, release time and other text fields are obtained by combining the target web page's a tag density cluster similarity with a text keyword filtering algorithm.
[0030] Optionally, POST page turning requests and GET page turning requests are identified through the page turning features of the target web page; the POST page turning request is an AJAX asynchronous request, and the list URL of the POST page turning request remains unchanged. The page turning rules based on the POST page turning request can be obtained by identifying text features such as "next page" and "next page"; the GET page turning request turns pages according to the changes in the page number of the list URL, such as xxx_1.html, xxx_2.html, and the page turning rules based on the GET page turning request can be identified through the numerical rules of page turning.
[0031] Optionally, text fields such as detail web page URL, list title and publication time are automatically identified based on list XPath rules, and automatic identification of article title, publication time, text and other contents is achieved through various technologies such as tag attribute features, tag density algorithm, keyword matching algorithm, etc., to determine the detail XPath rules.
[0032] Optionally, automatic rule recognition can be performed through technologies such as selenium, redis, jsoup, and document to reduce the workload of manually writing collection rules.
[0033] Specifically, after downloading the list HTML page of the seed information source website (ie, the public information source website), the list HTML page is used as the target web page.
[0034] S120: Determine an XPath list based on the a tag of the target web page.
[0035] After downloading the list HTML page of the seed information source website, you can determine the XPath list by the a tag of the target web page.
[0036] S130: Determine a list XPath rule based on the text length of the target webpage, preset keywords and the XPath list.
[0037] After the XPath list is obtained, the XPath rules of the list are automatically identified and filled based on the preset keywords, the XPath list and the text length of the target web page.
[0038] S140: Determine a page turning XPath rule.
[0039] S150: Acquire web page information based on the list XPath rule.
[0040] After obtaining the list XPath rule, the web page information is obtained based on the list XPath rule. The web page information includes the detail web page URL, the list title and the publishing time.
[0041] S160: Determine detailed XPath rules based on the web page information.
[0042] The rule recognition method provided in the embodiment of the present application obtains the target web page; determines the XPath list based on the a tag of the target web page; determines the list XPath rule based on the text length of the target web page, preset keywords and the XPath list; determines the page turning XPath rule; obtains web page information based on the list XPath rule; determines the detail XPath rule based on the web page information. Through the above method, the automatic recognition of XPath rules can be realized without manual operation, which can improve the recognition efficiency of XPath rules and avoid human errors.
[0043] In some embodiments, based on the a tag of the target web page, an XPath list is determined, including: based on the a tag of the target web page and the text in which the a tag is located, recursively obtaining the parent element of the a tag; determining each child element of the parent element; determining the hierarchical structure of each child element; and determining the XPath list based on the parent element, each child element, and the hierarchical structure of each child element.
[0044] See also Figure 3 , Figure 3 It is a flowchart of the identification list XPath rule provided in the embodiment of the present application.
[0045] like Figure 3 As shown, after downloading the list HTML page of the seed information source website, the list HTML page is used as the target web page.
[0046] Furthermore, based on the a tag of the target web page and the text where the a tag is located, the parent element of the a tag is recursively obtained.
[0047] It is understandable that there may be multiple a tags for a target web page.
[0048] Specifically, through JSOUP (JSOUP is a Java-based HTML parser that can directly parse a URL address and HTML text content), all a tags (denoted as Tags); get the href link of each a tag and the text it contains, that is, the text where each a tag is located; according to the a tag of the target web page and the text where the a tag is located, the XPath list can be reversed.
[0049] Among them, the a tag is a tag used to define a hyperlink in the HTML language, which can realize a link jump from one web page to another. The most important attribute of the a tag is the href attribute, which is used to specify the target of the link.
[0050] Furthermore, when the XPath list is obtained by inferring the a tag of the target web page and the text where the a tag is located, the parent element of each a tag needs to be recursively obtained.
[0051] For example, if a <book>The element is <library>The first child of an element, then <library>Elements are <book>The element's parent element.
[0052] Further, each child element of the parent element is determined.
[0053] Specifically, for each parent element of the a tag, each child element of the parent element can be located by an index.
[0054] The XPath language uses [n] as an index to select the nth child element under a parent element. For example, to select the first <book>element, use library / book[1], and the second element use library / book[2].
[0055] Furthermore, the hierarchical structure of each sub-element is determined.
[0056] Specifically, for each child element of the parent element of each a tag, the hierarchical structure of the child element needs to be determined.
[0057] Generally, in a web page, if an element is located in a multi-layer nested structure, " / " is used to separate different levels.
[0058] For example, if <book>The element is located <shelf>element, and <shelf>The element is <library>element, then the XPath path might be library / shelf / book[1].
[0059] Further, based on the parent element, each child element, and the hierarchical structure of each child element, an XPath list is determined.
[0060] Specifically, according to the parent element of each a tag, each child element of the parent element of each a tag, and the hierarchical structure of each child element, multiple complete XPath expressions can be combined, and the multiple complete XPath expressions can constitute an XPath list.
[0061] In some embodiments, based on the text length of the target webpage, preset keywords and the XPath list, a list XPath rule is determined, including: based on the text length of the target webpage and the preset keywords, filtering out matching XPath rules from the XPath list to generate an XPath rule set; generalizing the XPath rule set to obtain a generalized XPath rule set; counting XPath clusters in the generalized XPath rule set to obtain a counting result of each XPath cluster in the generalized XPath rule set, and sorting the counting results of each XPath cluster in sequence, extracting the XPath cluster with the largest counting result as the optimal XPath rule; based on the optimal XPath rule, searching for matching XPath rules in the XPath list, and combining and generating a list rule based on the matching XPath rules in the XPath list; based on the optimal XPath rule, searching for a target XPath rule whose class attribute and / or ID attribute meets the conditions, and replacing and combining the list rules based on the target XPath rule to generate a list XPath rule.
[0062] After the XPath list is obtained by reverse deduction, the matching XPath rules are filtered out from the XPath list based on the text length of the target web page and preset keywords to generate an XPath rule set.
[0063] Specifically, a variety of different keywords are pre-set for seed information source websites (i.e., public information source websites) (for example, the preset keywords may be "methods", "interpretation", "regulations", etc.), and the XPath rules corresponding to the a tags matching the preset keywords in the XPath list are retained; at the same time, the XPath rules corresponding to the a tags whose text length is greater than a preset threshold are retained.
[0064] The above-mentioned reserved XPath rules are the matching XPath rules filtered out in the XPath list. The reserved XPath rules are saved in the same set to obtain an XPath rule set.
[0065] Optionally, the XPath rule set can be saved using a LIST collection.
[0066] Furthermore, the XPath rule set is generalized to obtain a generalized XPath rule set.
[0067] Specifically, the method of generalizing the XPath rule set is as follows: Traverse the XPath rule set (i.e., LIST set), and reverse the process for each XPath rule in the LIST set and the next XPath rule corresponding to the XPath rule: First, delete the XPath path of the a tag. For example, there are two XPath paths of the a tag, which are recorded as / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul / li[1] / a and / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul / li[2] / a. After deleting the XPath path of the a tag, the two XPath paths obtained can be recorded as / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul / li[1] and / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul / li[2]; further, the XPath path for deleting the a tag is extracted upwards level by level according to the symbol " / " until the two XPath paths are repeated, and a generalized XPath rule set is obtained, which includes the XPath cluster / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul.
[0068] Furthermore, the XPath clusters in the generalized XPath rule set are counted to obtain the counting result of each XPath cluster in the generalized XPath rule set, and the counting results of each XPath cluster are sorted in sequence, and the XPath cluster with the largest counting result is extracted as the optimal XPath rule.
[0069] Specifically, the generalized XPath rule set includes multiple XPath clusters, but some XPath clusters may be repeated. In this case, each XPath cluster in the generalized XPath rule set may be counted to obtain a counting result of each XPath cluster in the generalized XPath rule set.
[0070] For example, the XPath cluster / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul may appear 10 times in the generalized XPath rule set, and the count result of the XPath cluster is 10.
[0071] After obtaining the counting result of each XPath cluster in the generalized XPath rule set, the counting result of each XPath cluster is sorted in order from large to small or from small to large, and the XPath cluster with the largest counting result is extracted as the optimal XPath rule.
[0072] Furthermore, based on the optimal XPath rule, a matching XPath rule is searched in the XPath list, and based on the matching XPath rules in the XPath list, a new XPath rule set is combined, and the new XPath rule set is the list rule.
[0073] Further, based on the optimal XPath rule, a target XPath rule whose class attribute and / or ID attribute meets the conditions is searched, and based on the target XPath rule, the list rules are replaced and combined to generate a list XPath rule.
[0074] Specifically, after obtaining the list rule, the list rule needs to be optimized: according to the obtained optimal XPath rule, find the target XPath rule whose class attribute and / or ID attribute meet the conditions, for example, try to find the XPath rule whose class attribute and ID attribute are accurate as the target XPath rule, and according to the target XPath rule, replace and combine the list rules to generate the list XPath rule.
[0075] For example, replace the target XPath rule / html / body / div[2] / div[4] / div[2] / div[2] / div[2] / div[2] / div / div[3] / ul with / / *[@class='news-list'] to generate a list XPath rule.
[0076] Furthermore, based on the list XPath rule, web page information is obtained, and the web page information includes the detail web page URL, list title and publishing time.
[0077] Optionally, store the list XPath rules to a Redis database.
[0078] In some embodiments, the page turning XPath rule includes a first page turning rule and a second page turning rule, the first page turning rule is a page turning rule based on a GET request, and the second page turning rule is a page turning rule based on a POST request.
[0079] Determine the page turning XPath rule, including: determine the tag to be identified that contains consecutive numbers in the target web page; determine whether the tag to be identified is an a tag; if the tag to be identified is an a tag, obtain the URL of the tag to be identified, and determine the first page turning rule based on the URL of the tag to be identified; if the tag to be identified is not an a tag, determine the second page turning rule based on the tag attribute characteristics of the tag to be identified.
[0080] The tag and attribute characteristics of the target web page can be used to identify what kind of request the list page turning request is (for example, POST request or GET request). The rule characteristic of the POST page turning request is that the page is turned by clicking buttons such as "Next Page" and "Next Page", and the browser URL does not change. The rule characteristic of the GET page turning request is that the browser's access URL changes, for example, xxx_1.html and xxx_2.html are different. The page turning rule based on the GET request (that is, the first page turning rule) is an XPath expression, such as / / *[text()='Next Page']. The page turning rule based on the POST request is page changes, such as xxx_${pageNo}.html, where ${pageNo} can be dynamically replaced with the page number during the crawling process.
[0081] See also< / library> < / shelf> < / shelf> < / book> < / book> < / book> < / library> < / library> < / book> Figure 4 , Figure 4 It is a flowchart of identifying page turning XPath rules provided in an embodiment of the present application.
[0082] like Figure 4 As shown, first obtain the target web page, which uses HTML language, and then obtain the page turning XPath rule according to the page number rule in the target web page.
[0083] Specifically, first determine the tags to be identified that contain consecutive numbers in the target web page, that is, find the tags that contain consecutive numbers such as 1, 2, 3, 4 in the tag text as the tags to be identified. Because the consecutive numbers 1, 2, 3, 4 contained in the tag text are usually page numbers, the page turning rules can be analyzed and extracted.
[0084] The tag to be identified is a special tag for page turning. The page turning XPath rule can be inferred based on the special tag for page turning.
[0085] Specifically, it is determined whether the tag to be identified is tag a.
[0086] If the tag to be identified is tag a, obtain the URL of the tag to be identified; extract the public string subsequences S1, S3 and the non-public string subsequence S2 from the URL of the tag to be identified, and determine whether S2 is a continuous number. If it is a continuous number, it meets the page turning rule based on the GET request, and the first page turning rule is S1${pageNo}S3.
[0087] For example, the three tags to be identified are all a tags, the URL of the first tag to be identified is recorded as https: / / www.shaoyang.gov.cn / shaoyang / gfxwjjd / nzcjd 2.shtml, the URL of the second tag to be identified is recorded as https: / / www.shaoyang.gov.cn / shaoyang / gfxwjjd / nzcjd 3.shtml, and the last URL is recorded as https: / / www.shaoyang.gov.cn / shaoyang / gfxwjjd / nzcjd 4.shtml. Then, the common string subsequence S1, i.e., https: / / www.shaoyang.gov.cn / shaoyang / gfxwjjd / nzcjd, and the common string subsequence S3, i.e., .shtml, can be extracted from the URLs of the three tags to be identified. The non-common string subsequence is the consecutive numbers 2, 3, and 4, and the first page turning rule is S1${pageNo}S3.
[0088] If the tag to be identified is not an a tag, or the tag to be identified is an a tag that does not contain an href attribute, it complies with the page turning rule based on the POST request; at this time, the second page turning rule can be determined based on the tag attribute characteristics of the tag to be identified: a page turning feature array is constructed, and the array contains preset page turning rules, such as: "next page", "next", "next page", ">", etc. By traversing the tag attribute characteristics of the tag to be identified in the target web page, the tag attribute containing the feature array value can be obtained, that is, the second page turning rule is XPATH= / / *[text()="tag attribute"].
[0089] Optionally, the first page turning rule and the second page turning rule are stored in a Redis database.
[0090] In some embodiments, the web page information includes a detail web page URL, a list title, and a publishing time.
[0091] Based on the list XPath rule, web page information is obtained, including: based on the list XPath rule, obtaining the URL of the detailed web page; using the text of the a tag where the list XPath rule is located as the list title; obtaining all the brother elements of the a tag where the list XPath rule is located and the text of each brother element; based on each brother element and the text of each brother element, obtaining the release time through the time regular expression.
[0092] In some embodiments, based on the web page information, the details XPath rules are determined, including: based on the details web page URL, downloading the details web page; based on the list title, determining the title element position in the details web page; based on the release time, determining the release time position in the details web page; based on the title element position and the release time position, obtaining the first text element in the details web page; based on the text tag attribute characteristics, determining the second text element in the details web page; based on the tag density algorithm, determining the third text element in the details web page; based on preset business keywords, determining the fourth text element in the details web page; based on the first text element, the second text element, the third text element and the fourth text element, determining the details XPath rules.
[0093] See also Figure 5 , Figure 5 It is a flowchart of the identification detail XPath rule provided in the embodiment of the present application.
[0094] like Figure 5 As shown, the identification of the details XPath rule requires the use of the list XPath rule to obtain the details webpage URL, list title and release time, and download the details webpage according to the details webpage URL. The details webpage adopts HTML language.
[0095] Furthermore, based on the list title, the position of the title element in the detail page is determined as follows: (1) Find the tags h1, h2, ... h6 with title characteristics and the elements whose title attributes include title on the detail page.
[0096] (2) Perform keyword matching between the text of the above elements and the list title, and take the label containing the text with the highest keyword matching degree as the title element.
[0097] (3) If the title element cannot be found or matched, traverse the full-text tags in the detail page and use the text keyword matching algorithm to match the tag that best matches the list title. The algorithm is as follows: convert the page into a document (i.e., document object); obtain all the tags in the document and use the text keyword matching algorithm to match the tag that best matches the list title as the title element; at the same time, to prevent over-recognition, additional judgment is required based on the position of the tag in the entire page. If the location of the tag obtained through selenium's webElement.getLocation is at the top of the page, the tag is used as the title element.
[0098] (4) Based on the title element located in the above steps, the title XPath rule of the details page can be inferred.
[0099] Preferably, high-precision rules such as class attributes or ID attributes are used for reverse deduction, and secondly, XPath paths are used for reverse deduction.
[0100] Further, based on the publishing time, the publishing time position in the details webpage is determined, the steps are as follows: (1) Find time tags with time features on the detail web page, that is, tags whose tag attributes contain time, date, and info, such as class=time, id=time, calss=date, id=date, etc.
[0101] (2) Match the text of the above labels with the release time, and take the element containing the text with the highest matching degree as the release time label.
[0102] (3) If the release time tag cannot be identified in the above two steps, the full-text tags in the details web page are traversed for text matching. For example, the element text contains elements such as "release time", "release date", "time", "date", etc., which are used as release time tags.
[0103] (4) Based on the release time tag located in the above steps, infer the time XPath rule of the details page.
[0104] Preferably, high-precision rules such as class attributes or ID attributes are used for reverse deduction, and secondly, XPath paths are used for reverse deduction.
[0105] Furthermore, the text elements are identified, and the text elements are located in multiple dimensions using methods such as the identified title element position and release time position, text tag attribute characteristics, tag density algorithm, preset business keywords, etc.
[0106] Specifically, based on the title element position and the publication time position, the first text element in the details web page is obtained: reverse the process upward step by step through the title element position and the publication time position. Each time you reverse the process upward, you need to obtain the text content of the element until the length of the obtained text accounts for 50%-90% of the entire HTML page content, then stop and take the non-tag element and non-publication time element sub-elements of the current element as the first text element.
[0107] Furthermore, based on the text tag attribute characteristics, the second text element in the details webpage is determined: through the text tag attribute characteristics (such as content, art_con, detail, etc.), the tag containing the text tag attribute characteristics in the details webpage is used as the second text element.
[0108] Further, based on the tag density algorithm, the third text element in the details page is determined: the text element is generally used Tags and Tags, find the continuous upper and lower levels in the text tags on the detail page Tag cluster or Tag cluster, take the parent element with the highest tag density as the third body element.
[0109] Furthermore, based on preset business keywords, the fourth text element in the details web page is determined: for example, the text of the bidding data must contain business keywords such as project name, tenderer, bidding deadline, etc. Different business keywords are preset according to different source websites. Based on the preset business keywords, all HTML tags in the details web page are traversed, and text recognition is performed step by step downward. The text elements containing the preset business keywords are recorded, and then the text length is compared to find the element feature with the smallest text length as the fourth text element.
[0110] Furthermore, based on the first body element, the second body element, the third body element and the fourth body element, the detailed XPath rules are reversely deduced.
[0111] Understandably, if Figure 5 As shown, if the first text element in the details webpage is obtained based on the title element position and the release time position, the details XPath rule can also be directly reversed based on the first text element. When no text element is matched based on the title element position and the release time position, subsequent identification can be performed.
[0112] Preferably, in order to improve the accuracy of text element identification and ensure that no text element is missed, four identification methods are required to jointly identify the first text element, the second text element, the third text element and the fourth text element, and then based on the first text element, the second text element, the third text element and the fourth text element, the detailed XPath rules are reversed.
[0113] Optionally, store the detailed XPath rules to a Redis database.
[0114] The rule recognition method provided in the embodiment of the present application improves the rule configuration part of the prior art and has the following three advantages: (1) by automatically recognizing list XPath rules, page turning XPath rules and detail XPath rules, the time-consuming problem of manually configuring XPath rules is reduced, and the degree of automation of rule configuration and data collection is improved; (2) the recognition method based on tag features can better adapt to the structural changes of different websites, solve the problems of manual dependence and difficult maintenance, and improve the adaptability and scalability of the rules; (3) combined with natural language processing technologies such as text similarity, the recognition algorithm can be continuously optimized and improved, the accuracy and efficiency of recognition can be improved, and the errors caused by manual participation can be reduced.
[0115] The present application also provides a rule recognition device. Figure 6 , Figure 6 6 is a schematic diagram of the structure of the rule identification device provided in the embodiment of the present application. In the embodiment of the present application, the rule identification device includes an acquisition module 610, a list rule determination module 620, a page turning rule determination module 630 and a detail rule determination module 640.
[0116] The acquisition module 610 is used to acquire the target web page.
[0117] The list rule determination module 620 is used to determine the XPath list based on the a tag of the target webpage; and determine the list XPath rule based on the text length of the target webpage, preset keywords and the XPath list.
[0118] The page turning rule determination module 630 is used to determine the page turning XPath rule.
[0119] The detail rule determination module 640 is used to obtain web page information based on the list XPath rule and determine the detail XPath rule based on the web page information.
[0120] In some embodiments, the list rule determination module 620 is used to recursively obtain the parent element of the a tag based on the a tag of the target web page and the text where the a tag is located; determine each child element of the parent element; determine the hierarchical structure of each child element; and determine the XPath list based on the parent element, each child element, and the hierarchical structure of each child element.
[0121] In some embodiments, the list rule determination module 620 is used to filter out matching XPath rules from the XPath list based on the text length of the target web page and preset keywords to generate an XPath rule set; generalize the XPath rule set to obtain a generalized XPath rule set; count the XPath clusters in the generalized XPath rule set to obtain the counting result of each XPath cluster in the generalized XPath rule set, and sort the counting results of each XPath cluster in sequence, and extract the XPath cluster with the largest counting result as the optimal XPath rule; based on the optimal XPath rule, search for matching XPath rules in the XPath list, and combine and generate list rules based on the matching XPath rules in the XPath list; based on the optimal XPath rule, search for target XPath rules whose class attributes and / or ID attributes meet the conditions, and replace and combine the list rules based on the target XPath rules to generate list XPath rules.
[0122] In some embodiments, the page turning XPath rule includes a first page turning rule and a second page turning rule, the first page turning rule is a page turning rule based on a GET request, and the second page turning rule is a page turning rule based on a POST request.
[0123] The page turning rule determination module 630 is used to determine the tag to be identified that contains consecutive numbers in the target web page; determine whether the tag to be identified is an a tag; if the tag to be identified is an a tag, obtain the URL of the tag to be identified, and determine the first page turning rule based on the URL of the tag to be identified; if the tag to be identified is not an a tag, determine the second page turning rule based on the tag attribute characteristics of the tag to be identified.
[0124] In one embodiment, the web page information includes a detail web page URL, a list title, and a publishing time.
[0125] The detail rule determination module 640 is used to obtain the detail web page URL based on the list XPath rule; use the text of the a tag where the list XPath rule is located as the list title; obtain all the brother elements of the a tag where the list XPath rule is located and the text of each brother element; based on each brother element and the text of each brother element, obtain the release time through the time regular expression.
[0126] In one embodiment, the details rule determination module 640 is used to download the details webpage based on the details webpage URL; determine the title element position in the details webpage based on the list title; determine the publishing time position in the details webpage based on the publishing time; obtain the first text element in the details webpage based on the title element position and the publishing time position; determine the second text element in the details webpage based on the text tag attribute characteristics; determine the third text element in the details webpage based on the tag density algorithm; determine the fourth text element in the details webpage based on preset business keywords; determine the details XPath rule based on the first text element, the second text element, the third text element and the fourth text element.
[0127] The present application also provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the rule recognition method.
[0128] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0129] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the rule identification method provided by the above methods is implemented.
[0130] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the rule identification methods provided by the above methods.
[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A rule recognition method, characterized in that: include: Get the target web page; Determine an XPath list based on the a tag of the target web page; Determine a list XPath rule based on the text length of the target webpage, preset keywords and the XPath list; Determine the page turning XPath rule; Based on the XPath rules in the list, obtain web page information; Based on the web page information, a detailed XPath rule is determined.
2. The rule identification method according to claim 1, characterized in that: The step of determining an XPath list based on the a tag of the target webpage includes: Recursively obtain the parent element of the a tag based on the a tag of the target webpage and the text where the a tag is located; Determine each child element of the parent element; Determining the hierarchical structure of each of the sub-elements; Based on the parent element, each of the child elements, and the hierarchical structure of each of the child elements, an XPath list is determined.
3. The rule identification method according to claim 2, characterized in that: The determining of the list XPath rule based on the text length of the target webpage, the preset keywords and the XPath list includes: Based on the text length of the target webpage and preset keywords, filtering out matching XPath rules from the XPath list to generate an XPath rule set; Generalizing the XPath rule set to obtain a generalized XPath rule set; Counting the XPath clusters in the generalized XPath rule set to obtain a counting result of each XPath cluster in the generalized XPath rule set, and sequentially sorting the counting results of each XPath cluster, and extracting the XPath cluster with the largest counting result as the optimal XPath rule; Based on the optimal XPath rule, searching for a matching XPath rule in the XPath list, and combining and generating a list rule based on the matching XPath rules in the XPath list; Based on the optimal XPath rule, a target XPath rule whose class attribute and / or ID attribute meets the conditions is searched, and based on the target XPath rule, the list rules are replaced and combined to generate a list XPath rule.
4. The rule identification method according to claim 1, characterized in that: The page turning XPath rule includes a first page turning rule and a second page turning rule, wherein the first page turning rule is a page turning rule based on a GET request, and the second page turning rule is a page turning rule based on a POST request; The step of determining the page turning XPath rule includes: Determine the target web page's tags that contain consecutive numbers and need to be identified; Determine whether the tag to be identified is tag a; If the tag to be identified is tag a, obtaining the URL of the tag to be identified, and determining the first page turning rule based on the URL of the tag to be identified; If the tag to be identified is not tag a, the second page turning rule is determined based on the tag attribute characteristics of the tag to be identified.
5. The rule identification method according to claim 1, characterized in that: The web page information includes the URL of the detailed web page, the list title and the publishing time; The obtaining of web page information based on the XPath rule in the list includes: Based on the XPath rule in the list, obtain the URL of the detail webpage; The text of the a tag where the list XPath rule is located is used as the list title; Get all the sibling elements of the a tag where the XPath rule in the list is located and the text of each of the sibling elements; Based on each of the sibling elements and the text of each of the sibling elements, the publishing time is obtained through a time regular expression.
6. The rule identification method according to claim 5, characterized in that: The determining of the detailed XPath rule based on the webpage information includes: Based on the details webpage URL, download the details webpage; Based on the list title, determining the position of the title element in the detail webpage; Based on the publishing time, determining the publishing time position in the details webpage; Based on the position of the title element and the position of the publishing time, obtaining the first text element in the details webpage; Determining a second text element in the details webpage based on the text tag attribute characteristics; Determining a third text element in the details webpage based on a tag density algorithm; Based on the preset business keywords, determining a fourth text element in the details webpage; A detail XPath rule is determined based on the first body element, the second body element, the third body element, and the fourth body element.
7. A rule recognition device, characterized in that: include: An acquisition module, used to acquire a target web page; A list rule determination module, used to determine an XPath list based on the a tag of the target web page; Determine a list XPath rule based on the text length of the target webpage, preset keywords and the XPath list; A page turning rule determination module is used to determine the page turning XPath rule; A detail rule determination module, used to obtain web page information based on the XPath rule in the list; Based on the web page information, a detailed XPath rule is determined.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the rule identification method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the rule identification method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the rule identification method according to any one of claims 1 to 6 is implemented.