A crawler method for a subject portal website
By employing regular expressions and Bloom filters to identify parent and child page links in a web crawler system, and combining the Heritrix and Webmagic frameworks, the problem of duplicate downloads in the web crawler system was solved, achieving efficient information extraction and deduplication, and improving the accuracy of information acquisition and system performance.
Patent Information
- Application Number
- CN202110051839.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-01-15
AI Technical Summary
Existing web crawler systems tend to waste CPU resources and increase database access load when crawling web pages with the same content, and it is difficult to effectively prevent duplicate downloads.
We employ a theme-based portal website crawler approach, using regular expressions to identify parent and child page links. We combine Bloom filters with the Heritrix and Webmagic frameworks, utilizing Spring and Hibernate technologies to design an efficient deduplication strategy, including webpage content extraction, persistent data storage, and incremental crawling. We also use XPath syntax and Bloom filters to prevent duplicate extractions.
It enables vertical searching of information, quickly detects website updates, extracts only valid information, reduces duplicate downloads, avoids wasting CPU resources, reduces database load, and improves the accuracy and efficiency of information retrieval.
Smart Images

Figure CN112749315B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of network information crawling, in particular to a theme portal website crawler method. Background Art
[0002] In the open environment of the Internet, the explosive growth of shared network information has provided people with a large amount of information resources. However, this has also brought huge challenges. There are many types of information, and it is becoming increasingly difficult to effectively collect and use this information. At this time, search engines began to be born. Searching network information through keywords greatly facilitates people to search for information effectively and can meet most information needs. However, most search engines are mainly horizontal searches. The main disadvantage of this method is that the search results returned are of low accuracy and contain a lot of interference information. With the development of information diversification, this search strategy can no longer meet the specific needs of users.
[0003] For this purpose, web theme crawlers came into being. Web theme crawlers only crawl the required information, which can not only greatly reduce time consumption, but also obtain updated content in time, and the information obtained will be more accurate and comprehensive, which greatly reduces interference information. However, an important problem that the web theme crawler system needs to solve now is to prevent web pages containing the same content from being downloaded multiple times, avoid wasting a lot of CPU resources, and reduce the load brought by database access. Summary of the invention
[0004] In order to solve the problem that web pages containing the same content are downloaded multiple times when the existing network theme crawler system crawls them once, which wastes a lot of CPU resources and increases the load on database access, the present invention deduplicates the two links of content crawling and incremental updating in the theme portal website crawler system, and proposes an efficient deduplication strategy, which is superior to traditional methods in performance and scalability.
[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a theme portal website crawler method, comprising:
[0006] Web page link analysis and extraction: Design regular expressions based on the theme website to identify parent page and child page links, and determine whether the page belongs to the link within the theme website. Only the links within the theme website are processed. If it is identified as a parent page, the child page links in the parent page are extracted. If it is identified as a child page, the text content of the child page is extracted.
[0007] Web page content extraction: extract the text content under the sub-page link, store the extracted text content in a static class, and exit after successful extraction;
[0008] Data persistence storage: used to store the text content extracted from each sub-page link;
[0009] Incremental crawling: Crawl the updated content in the theme website. During each incremental update, re-extract the links to the theme website’s homepage and only process the new links.
[0010] Furthermore, in the web page content extraction, for all the text formats found on the web page, first find a format, extract the text from all pages, and then store it in the database. At this time, the text fields of different formats are empty in the database, and then reversely check its format from the link of the empty text character, write the corresponding extraction code, and then loop several times to find all the formats, so as to design the corresponding xpath syntax, save all xpath formats in a list, extract the text content by traversing the list, exit the loop after successful extraction, and define the extraction process as a static class.
[0011] Furthermore, in the incremental crawling, by inputting a seed link, it is determined whether the seed link is a parent page. If it is a parent page, a new child page link is extracted from the parent page and placed in a queue to be crawled, the program ends, and then the web page content is crawled; if it is not a parent page, it is a child page. At this time, it is determined whether the main text content of the child page link has been extracted. If it has been extracted, the program ends; if it has not been extracted, the link is placed in a queue to be crawled, the main text content under the link is extracted, the extracted content is persistently stored as data, and the program ends.
[0012] Furthermore, in the incremental crawling, a Bloom filter is used to screen and determine the incrementally updated website links.
[0013] Furthermore, the Bloom filter uses a BitSet function and a hash function in combination to judge the incrementally updated website links, wherein the BitSet function is defined as a static private variable.
[0014] Furthermore, in the BitSet function, Path is used as the path to be loaded and saved by BitSet; the getBitSet method loads and deserializes the BitSet object from the Path path. If there is no object that can be loaded, false is returned, a new instance is regenerated, and finally the object stream is directly called to serialize and save it; the hash function calculates each character in the web link, adds each result, and then takes the modulus with the length of the BitSet, so that the hash distribution can be more uniform.
[0015] Furthermore, the hash function first selects a seed, and the seed is selected as a prime number. A total of 8 hash values are set in the Bloom filter. By passing in 8 different hash seeds, different hash values can be obtained. When a link is passed in, the add(String value) method of the Bloom filter is called to first determine whether the value is empty. If it is not empty, the addValue(String value) method is called, and addValue is defined as a static variable.
[0016] Furthermore, the Bloom filter uses the contains method to determine whether there is an identical URL link. When a link is passed in, all hash methods are called for determination. If a hash algorithm value is not false, it can be determined that the URL is not repeated, and then the content is extracted.
[0017] Furthermore, in the data persistent storage, after the web page information is extracted, it can be persistently stored in the database as needed, or saved in different formats, using the page.putFiled(String key,Objectfield) method to store the extracted text content.
[0018] Furthermore, in the regular expression for analyzing and extracting the web page links, the regular expression used to determine whether the extracted link is a parent page or a child page is: http: / / www\\.(.*\\.)? agri\\.cn / .*(htm)$; the regular expression used to extract all parent pages and child pages is: http: / / www\\.(.*\\.)? agri\\.cn / .+, and this regular expression can be used to determine whether the link belongs to the subject website.
[0019] Technical effects and advantages of the present invention:
[0020] (1) For a certain topic or website, the links between parent pages and child pages are used to extract information to achieve vertical search;
[0021] (2) Ability to continuously and quickly detect website updates and download them;
[0022] (3) Only extract valid information such as pictures and web page content, organize and save them, and facilitate information analysis;
[0023] (4) In web page content extraction, for all text formats found on the web page, first find a format, extract the text from all pages, and then store it in the database. At this time, the text fields of different formats are empty in the database. Then reversely check the format from the link of the empty text character, write the corresponding extraction code, and loop several times to find all formats, so as to design the corresponding xpath syntax, save all xpath formats in a list, extract the text content by traversing the list, exit the loop after successful extraction, and define the extraction process as a static class, which is more efficient and simpler than the commonly used methods in the past, and the content obtained is almost non-duplicate;
[0024] (5) In incremental crawling, by inputting a seed link, it is determined whether the seed link is a parent page. If it is a parent page, a new sub-page link is extracted from the parent page and placed in a queue to be crawled. The program ends, and then the web page content is crawled. If it is not a parent page, it is a sub-page. At this time, it is determined whether the text content of the sub-page link has been extracted. If it has been extracted, the program ends. If it has not been extracted, the link is placed in a queue to be crawled, and the text content under the link is extracted. The extracted content is stored persistently, and the program ends. This method can quickly extract incremental updates and effectively prevent repeated extraction.
[0025] (6) The present invention combines two Heritrix frameworks with the Webmagic framework. In order to facilitate deployment and use, Spring and Hibernate technologies are combined. Spring mainly uses its IOC module and MVC module. By integrating with Hibernate, the system has a certain flexibility.
[0026] (7) Each character of the link is calculated through the hash function in the Bloom filter, and then the results are added and modulo the length of the BitSet. This can make the hash distribution more uniform, and the selected hash seed can determine whether the URL is repeated, achieving efficient deduplication;
[0027] The pages obtained by this crawler program are almost non-duplicate and can accurately obtain the required topics. It can also effectively prevent web pages containing the same content from being downloaded multiple times, avoid wasting a lot of CPU resources, and reduce the load caused by database access. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a flow chart of webpage content crawling of the present invention;
[0029] Figure 2 It is a flow chart of the incremental crawling method of the present invention;
[0030] Figure 3 A webpage content crawling result display in one embodiment of the present invention;
[0031] Figure 4 Performing web page query display in one embodiment of the present invention;
[0032] Figure 5 The content captured by the input URL is displayed in one embodiment of the present invention. DETAILED DESCRIPTION
[0033] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0034] Example:
[0035] like Figure 1-5 As shown, a subject portal crawler method includes:
[0036] Web page link analysis and extraction: Design a regular expression based on the theme website to identify the parent page and child page links, and determine whether the page belongs to the link within the theme website. Only the links within the theme website are processed. If it is identified as a parent page, the child page link in the parent page is extracted. If it is identified as a child page, the text content of the child page is extracted. The regular expression used to determine whether the extracted link is a parent page or a child page is: http: / / www\\.(.*\\.)? agri\\.cn / .*(htm)$; the regular expression used to extract all parent pages and child pages is: http: / / www\\.(.*\\.)? agri\\.cn / .+, and this regular expression can be used to determine whether it belongs to the link within the theme website;
[0037] Web page content extraction: extract the text content under the sub-page link, and store the extracted text content in a static class. Exit after successful extraction. In web page content extraction, for all text formats found under the web page, first find a format, extract the text from all pages, and then store it in the database. At this time, the text fields of different formats are empty in the database. Then reversely check the format from the link of the empty text character, write the corresponding extraction code, and loop several times to find all formats, so as to design the corresponding xpath syntax, save all xpath formats in a list, extract the text content by traversing the list, exit the loop after successful extraction, and define the extraction process as a static class;
[0038] Data persistence storage: used to store the text content extracted from each sub-page link. After the web page information is extracted, it can be persisted in the database or saved in different formats as needed. The page.putFiled(String key,Object field) method is used to store the extracted text content.
[0039] Incremental crawling: crawl the updated content in the theme website. Each time an incremental update occurs, re-extract the link to the theme website homepage and only process the new link. In the incremental crawling, by inputting the seed link, determine whether the seed link is the parent page. If it is the parent page, extract the new child page link from the parent page and put it in the queue to be crawled. The program ends and then crawl the web page content. If it is not the parent page, it is a child page. At this time, determine whether the body content of the child page link has been extracted. If it has been extracted, end the program. If it has not been extracted, put the link into the queue to be crawled and extract the body content under the link. The extracted content is stored persistently and the program ends. Use Bloom filter to screen and judge the incremental updated URL link. The BitSet function and hash function are used in Bloom filter to judge the incremental updated URL link. The BitSet function is defined as a static private variable. In the BitSet function, Path is used as the path to be loaded and saved by BitSet. The getBitSet method loads and deserializes the BitSet object from the Path path. If there is no object to be loaded, false is returned, a new instance is regenerated, and the object stream is directly called to serialize and save it. The hash function calculates each character in the web link, adds each result, and then modulosizes it with the length of BitSet, which can make the hash distribution more uniform. The hash function first selects a seed, which is a prime number. A total of 8 hash values are set in the Bloom filter. By passing in 8 different hash seeds, different hash values can be obtained. When a link is passed in, the add(String value) method of the Bloom filter is called to first determine whether the value is empty. If it is not empty, the addValue(String value) method is called again, and addValue is defined as a static variable. The contains method is used in the Bloom filter to determine whether there is an identical URL link. When a link is passed in, all hash methods are called for judgment. If the value of a hash algorithm is not false, it can be determined that the url is not repeated, and then the content is extracted.
[0040] Take crawling "China Agricultural Information Network" as an example. This is a comprehensive agricultural portal website with many categories. This crawler crawled data for three sections and clustered them into three categories through text clustering with a high accuracy rate, proving that this crawler can crawl relatively clean data from large portal websites.
[0041] (1) Distinguishing between parent page and child page
[0042] The link of the parent page of China Agricultural Information Network is named with English letters plus " / ", such as the link of market classification is http: / / www.agri.cn / V20 / SC / , while the link of sub-page is a number plus the suffix ".htm".
[0043] After the link naming rules are determined, the parent page and child page links can be identified through regular expressions. Since Webmagic uses multi-threaded crawling, the regular expression should be modified with final to ensure thread safety.
[0044] Here is a regular expression that matches links:
[0045] public static final String URL_POST="http: / / www\\.(.*\\.)?agri\\.cn / .*(htm)$";
[0046] public static final String URL="http: / / www\\.(.*\\.)?agri\\.cn / .+";
[0047] URL_POST means matching the code starting with "http: / / www.agri.cn / " and ending with "htm". The "(.*\\.)?" is to match the link similar to "http: / / www.flower.agri.cn / ".
[0048] The URL extracts all links belonging to the China Agricultural Information Network, including parent pages and child pages. This regular expression can be used to accurately determine whether it is an internal link, preventing crawlers from crawling irrelevant web pages and falling into the black hole of the Internet.
[0049] AgriInfoTest implements the PageProcessor interface and the public void process(Pagepage){} method. The Page object contains the HTML page code, which is used to extract valid information from it. The Page.getUrl() method returns the link of the page. At this time, it is necessary to determine whether the page is a parent page or a child page, and call the Java regular expression:
[0050] page.getUrl().regex(URL_POST).match();
[0051] If it is a child page, you can extract the content. After extracting all the links from the parent page, you can skip it directly without extracting information. Call page.setSkip(true) to skip the current page without persisting it.
[0052] (2) Web page content extraction
[0053] After obtaining all the formats of the web page, it is found that there are 8 types of text formats in the China Agricultural Information Network, and the xpath syntax is as follows:
[0054] ① / / div[@id=\"TRS_AUTOADD\"] / allText()
[0055] ② / / div[@class=\"nr_m16\"] / allText()
[0056] ③ / / td[@class=\"hui_14-1\"] / text()
[0057] ④ / / div[@class=\"TRS_Editor\"] / allText()
[0058] ⑤ / / td[@class=\"hui_14-1\"] / p / allText()
[0059] ⑥ / / div[@class=\"de_tag_con\"] / p / allText()
[0060] ⑦ / / div[@class=\"TRS_PreAppend\"] / p / allText()
[0061] ⑧ / / table / allText()
[0062] Since the text formats overlap, it is possible to match two XPath syntaxes at the same time, such as formats ③ and ⑤. If the XPath OR operation is called, the captured text will be repeated in two sections. At this time, an algorithm needs to be called to determine whether the text is repeated, and then remove the repeated parts of the text after it has been repeated several times.
[0063] In order to avoid multiple judgments on the extraction results, we can use multiple nested ifs to judge when the current format is not matched successfully.<string>Object, at this time, to determine whether the text is captured, you cannot simply call the list.isEmpty() method to judge, because some web pages will return spaces when the text cannot be extracted. At this time, the List will have space objects, so it cannot be judged. Therefore, the list.toString() method is used to return the data content inside, and the formal expression is used to match whether the data is empty or space. The corresponding regular expression code is contentList.toString().matches("\\[\\s*\\]"), "\\s" represents a space. At this time, the brackets that can be matched may have one or more spaces. When extracting content, you cannot know in advance which format will be extracted successfully. In order to successfully judge and save performance, a nested if statement is used for judgment.
[0064]
[0065] Therefore, all xpath formats are saved in a list in advance, and the text is extracted by traversing the list. When the text is successfully extracted, the loop is exited. The idea is very simple, and the performance and scalability are very high. The processing code is as follows:
[0066]
[0067] The static block initializes contentList when loading the class. The static block will only be executed once when the class is loaded, which can avoid initializing contentList every time it is used. contentList is designed as a static object, which can be shared by all instances of the class. The String object inside can be saved all the time and does not need to be reallocated every time it is called, saving space and time. The getContent method passes in the page object. Since the first seven formats are of the same type, the text can be directly traversed in the loop to extract the text. Once the extraction is successful, it will return immediately. If the current seven formats are not correct, the eighth format needs to be called for corresponding processing and then the result is returned. The performance is improved and the repetition rate has been reduced to 0.
[0068] (3) Data persistence storage
[0069] After all the information on the web page is extracted, putField(key, value) is called on the page to save the relevant data. Its structure is equivalent to a map, which can then be obtained in the pipeline through ResultItems for persistent storage.
[0070] (4) Incremental crawling
[0071] Since the content of the website is constantly updated, in order to obtain updated data in time, it is necessary to design an incremental crawling strategy. The incremental crawling requirement of this crawler is that even after the program is closed, when crawling data again, it can determine which ones have been crawled and which ones have not been crawled. When crawling general websites, by calculating whether the MD5 (MessageDigest Algorithm) value of the homepage has changed, it can be determined whether the webpage has been updated. However, for large comprehensive websites, this method is not suitable. Simply calculating the MD5 of the homepage can only determine whether the homepage has changed, but whether there is an update in the classified website under the China Agricultural Information Network, it cannot be perceived. It can only calculate the MD5 of all classified pages and then compare them. At this time, all homepages and their MD5 values need to be saved separately. The amount of data is large and very time-consuming. An effective strategy should be to save all sub-pages, and re-extract the links of the homepage every time an incremental update is made, and only the new links can be processed.
[0072] Based on the analysis of URL deduplication strategy, Bloom filter is used to filter URL. The core of Bloom filter is the bit array and the corresponding hash function. Java BitSet is a typical bit array, which can apply for the required bit vector on demand. Each bit is a boolean value. By default, all bit values are false. For the agricultural information network, it must have a proprietary BitSet that does not allow replacement and can provide methods for saving and loading for multiple uses. At this time, the singleton mode is needed to ensure that only one instance of the system exists.
[0073] There are many ways to implement the singleton pattern. The lazy style uses double if judgment to ensure thread safety and high efficiency. The hungry style directly obtains it through the static statement block. When the class is loaded, it directly gets the instance of the class, which can ensure that there is only one instance. The code structure is concise. Compared with the lazy loading, its disadvantage is that it will occupy system memory when it is not used at the beginning. On the crawler system, the Bloom filter is used when the program starts, so there is not much difference between the hungry style and the lazy style. For convenience, lazy loading is directly used.
[0074]
[0075]
[0076] BitSet is designed as a static private variable and does not provide external getter and setter methods. It can be guaranteed not to be modified by external classes and ensure uniqueness. Path is the path where BitSet is loaded and saved. When the web project is started, the deployment path of the project must be obtained so that it can be saved in the corresponding location as needed. The Spring framework allows the configuration of webAppRootKey in web.xml. By specifying the corresponding key, you can call the System.getProperty(key) method to obtain the project path and then select the saved location. The getBitSet method loads and deserializes the BitSet object from the path path. If there is no object to load, it returns false and regenerates a new instance. The saving method is simple. It can directly call the Java object stream to serialize and save it.
[0077] The url is presented in String form in Java, so the hash function is calculated for each character in it, and then the results are added and modulo the length of the BitSet, which can make the hash distribution more uniform.
[0078]
[0079]
[0080] The hash function first selects a seed, which is usually a prime number. This Bloom filter uses 8 hash values. You only need to pass in 8 different hash seeds to get different hash values. The randomly selected seeds are 3, 5, 7, 11, 13, 31, 37, and 61. Because the length of the bit array is 2^25, which is an integer power of 2, cpa is the length. The expression "(cap-1)&result" can quickly take the modulus of result and cap through AND operation.
[0081] When a link is passed in, the filter's add(String value) method is called. It first determines whether the value is empty. If it is not empty, the addValue(String value) method is called. Its implementation is as follows:
[0082]
[0083] The method is declared as static and can be called without instantiating an object, making it convenient to use as a tool class; func is an array of hash algorithms, and the object SimpleHash contains the hash algorithm and hash seed, which are encapsulated into a class for unified management, making the code clear and tidy; the contains method of the filter determines whether the same URL exists. When a link is passed in, all hash methods are called for judgment. If a hash algorithm value is not false, it can be determined that the URL is not repeated.
[0084]
[0085]
[0086] like Figure 3-Figure 5 As shown in the figure, after the program runs and crawls, the database has reached hundreds of thousands of web pages. Figure 3 These are some of the displays. Each paragraph contains the main content of a web page. When the mouse passes over it, the corresponding title and author will be displayed. Clicking on a paragraph will jump to the original link.
[0087] Referring to Table 1 below, the crawled web pages are clustered using the K-means method. It is found that the pages obtained by this crawler program have almost no duplication and can accurately obtain the required topics.
[0088] Table 1: Content display of text clustering of crawled content
[0089]
[0090] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.< / string>
Claims
1. A theme portal website crawler method, characterized in that: include Web page link analysis and extraction: Design regular expressions based on the theme website to identify parent page and child page links, and determine whether the page belongs to the link within the theme website. Only the links within the theme website are processed. If it is identified as a parent page, the child page links in the parent page are extracted. If it is identified as a child page, the text content of the child page is extracted. Web page content extraction: extract the text content under the sub-page link, store the extracted text content in a static class, and exit after successful extraction; Data persistence storage: used to store the text content extracted from each sub-page link; Incremental crawling: crawl the updated content in the theme website. Each time the incremental update occurs, re-extract the link of the theme website homepage and only process the new link. In the incremental crawling, the Bloom filter is used to screen and judge the incremental updated URL links. The Bloom filter uses the BitSet function and the hash function to judge the incremental updated URL links. The BitSet function is defined as a static private variable. In the BitSet function, Path is used as the path to be loaded and saved by BitSet. The getBitSet method loads and deserializes the Path path to obtain the BitSet object. If there is no object that can be loaded, false is returned, and a new instance is regenerated. Finally, the object stream is directly called to serialize and save it. The hash function calculates each character in the web link, adds each result, and then takes the modulus with the length of BitSet, which can make the hash distribution more uniform. The hash function first selects a seed, which is a prime number. A total of 8 hash values are set in the Bloom filter. By passing in 8 different hash seeds, different hash values can be obtained. When a link is passed in, the add(String value) method of the Bloom filter is called to first determine whether the value is empty. If it is not empty, the addValue(String value) method is called, and addValue is defined as a static variable.
2. A theme portal website crawler method according to claim 1, characterized in that: In the web page content extraction, for all the text formats found on the web page, first find a format, extract the text from all pages, and then store it in the database. At this time, the text fields of different formats are empty in the database, and then reversely check its format from the link of the empty text character, write the corresponding extraction code, and then loop several times to find all the formats, so as to design the corresponding xpath syntax, save all xpath formats in a list, extract the text content by traversing the list, exit the loop after successful extraction, and define the extraction process as a static class.
3. A theme portal website crawler method according to claim 1, characterized in that: In the incremental crawling, by inputting a seed link, it is determined whether the seed link is a parent page. If it is a parent page, a new child page link is extracted from the parent page and placed in a queue to be crawled, the program ends, and then the web page content is crawled; if it is not a parent page, it is a child page. At this time, it is determined whether the main text content of the child page link has been extracted. If it has been extracted, the program ends; if it has not been extracted, the link is placed in a queue to be crawled, the main text content under the link is extracted, the extracted content is persistently stored, and the program ends.
4. A theme portal website crawler method according to claim 1, characterized in that: The Bloom filter uses the contains method to determine whether there is an identical URL link. When a link is passed in, all hash methods are called for judgment. If the value of a hash algorithm is not false, it can be determined that the URL is not repeated, and then the content is extracted.
5. A theme portal website crawler method according to claim 1, characterized in that: In the data persistent storage, after the web page information is extracted, it can be persistently stored in the database as needed, or it can be saved in different formats, and the extracted text content can be stored using the page.putFiled(String key,Object field) method.
6. A theme portal website crawler method according to claim 1, characterized in that: In the regular expression for analyzing and extracting the web page links, the regular expression used to determine whether the extracted link is a parent page or a child page is: http: / / www\\.(.*\\.)? agri\\.cn / .*(htm)$; the regular expression used to extract all parent pages and child pages is: http: / / www\\.(.*\\.)? agri\\.cn / .+, and this regular expression can be used to determine whether it is a link within the subject website.
