Distributed data crawling methods based on HBase and MapReduce

Abstract

This invention relates to the field of computer application technology, specifically disclosing a distributed data crawling method based on HBase and MapReduce. The method includes: S100 creating a table in HBase, which contains three column families for storing the target URL, crawling and parsing flags, and the source code of the content crawled from the target URL; S200 collecting URLs with high out-degrees from the homepages of the target website as seed URLs and inserting them as new rows into the table described in step S100. This invention's distributed data crawling method based on HBase and MapReduce utilizes the MapReduce component to crawl data from the network in a distributed manner and stores the crawled data in an HBase database, which can reduce the time cost of obtaining unstructured data and improve the reliability of storing crawled data.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, specifically a distributed data crawling method based on HBase and MapReduce. Background Technology

[0002] With the rapid development of Internet technology and the advent of the big data era, the data on the World Wide Web is growing explosively, especially unstructured data, which accounts for more than 75% of it. This data contains a massive amount of useful information, and how to efficiently crawl the required unstructured data from the Internet has become a highly concerned issue.

[0003] Web crawlers are technologies that simulate browsers sending network requests and receiving responses to automatically retrieve information from the internet. If only a single machine is used for crawling, retrieving massive amounts of data from the internet would be extremely time-consuming.

[0004] MapReduce is a parallel distributed computing framework used for parallel computation of large-scale datasets; HBase is a distributed columnar database that provides high reliability, scalability, high performance, and real-time read / write capabilities. By using the MapReduce component to crawl data from the network in a distributed manner and storing the crawled data in an HBase database, the time cost of acquiring unstructured data can be reduced, and the reliability of storing crawled data can be improved.

[0005] In summary, this invention proposes a distributed data crawling method based on HBase and MapReduce. Summary of the Invention

[0006] The purpose of this invention is to provide a distributed data crawling method based on HBase and MapReduce, which can complete parallel distributed data crawling. It cleverly utilizes the shuffling mechanism between map and reduce in MapReduce to solve the URL deduplication problem in distributed crawling, avoids repeated crawling of crawled web pages, and saves the crawled data in the columnar database HBase, which is convenient for subsequent batch processing of data in the "doc" column family.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a distributed data crawling method based on HBase and MapReduce, the method comprising:

[0008] S100 creates a table in HBase containing three column families, which are used to store the target URL, the crawling and parsing flags, and the source code of the content crawled from the target URL.

[0009] S200 collects URLs with large out-degree values ​​from the homepages of modules on the target website as seed URLs and inserts them as new rows into the table described in step S100.

[0010] S300 inputs the ur l column and crawl flag column of each row of the table described in step S200 into the map task function of the crawling job;

[0011] S400 inputs the URL column, parsing flag column, and content source code column of each row of the table updated in step S300 into the map task function of the parsing job; the map task function constructs an output key-value pair for each parsed URL, uses the new URL as the key of the output key-value pair, and the value of the key-value pair is "fa l se"; the key-value pair output by the map end is input into the combin function of the parsing job.

[0012] S500 inputs the output of the combine function described in step S400 into the reduce function of the parsing job;

[0013] S600 iteratively executes steps S300 to S500, incrementally crawling the target website data until the target data volume is reached or the target website is completely crawled.

[0014] In a preferred embodiment of the present invention, in step S100: the crawling and parsing flag column family includes two columns, which record the crawling flag and the parsing flag respectively. The table uses the hash value of each row ur l as the row key.

[0015] In a preferred embodiment of the present invention, in step S200: the crawling and parsing flags of the new row are set to not crawled and not parsed, respectively, and the content source code column is empty.

[0016] In a preferred embodiment of the present invention, in step S300, the map function determines if a certain row in the table is not crawled by the crawling flag column. If so, it uses the HttpURLConnection tool library to crawl the URL content source code of that row. If the crawling is successful, the crawled URL source code is saved in the content source code column family of that row in the table, and the crawling flag column of that row is set to crawled. If the length of the crawling result is less than 100 or the crawling fails, no changes are made.

[0017] In a preferred embodiment of the present invention, in step S400, the map function constructs output key-value pairs for each input row. For an input row, the value of the target URL column in that row is used as the key of the output key-value pair. If the content source code column of that row is empty, the value of the corresponding output key-value pair is "true", otherwise it is "true". If the content source code column of the input row is not empty and the parsing flag column is unparsed, the map function uses a customized regularization rule to parse the content source code of that row. The combine function iterates through the values ​​of all key-value pairs with the same URL as the key. If there is a "true" value among the values ​​of all key-value pairs with a certain URL as the key, a key-value pair is output for that URL.<ur l,“true”> Otherwise, output a key-value pair for that URL.<ur l,“fa l se”> .

[0018] As a preferred embodiment of the present invention, the backend searches for all cloud printing desktops that support the printer model and dynamically allocates the most idle resources to the user side.

[0019] As a preferred embodiment of the present invention, the operation logic of the reduce function in step S500 is as follows: if there is a "true" value in the value list of key-value pairs with the same ur l as the key, then set the parsing flag column of the corresponding row in the table updated in step S400 to be parsed; otherwise, insert a new row in the table, with the ur l column containing the ur l value, and set the crawling and parsing flags to not crawled and not parsed.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] This invention utilizes the MapReduce component to crawl data from the network in a distributed manner and stores the crawled data in an HBase database, which can reduce the time cost of obtaining unstructured data and improve the reliability of storing crawled data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0023] Figure 1 This is a flowchart illustrating the overall process of the distributed data crawling method based on HBase and MapReduce in this invention. Detailed Implementation

[0024] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0025] Please see Figure 1 This invention provides a distributed data crawling method based on HBase and MapReduce, the method comprising:

[0026] Step S100 creates a table in HBase containing three column families, which are used to store the target URL, the crawling and parsing flags, and the source code of the content crawled from the target URL. The crawling and parsing flags column family contains two columns, which record the crawling flags and the parsing flags respectively. The table uses the hash value of each URL as the row key to make the partition more even and thus effectively prevent hotspot effects.

[0027] S200 collects URLs with large out-degrees, such as the homepage of the target website, as seed URLs and inserts them as new rows into the table described in step S100. The crawling and parsing flags of the new rows are set to "not crawled" and "not parsed" respectively, and the content source code column is empty.

[0028] S300 inputs the URL column and crawl flag column of each row in the table described in step S200 into the map task function of the crawling job. The map function determines if the crawl flag column of a row in the table is not crawled, and then uses the HttpURLConnection utility library to crawl the source code of the URL of that row. If the crawling is successful, the crawled URL source code is saved in the content source code column family of that row in the table, and the crawl flag column of that row is set to crawled; if the length of the crawled result is less than 100 or the crawling fails, no changes are made. This step uses crawl flags to avoid repeatedly crawling the same URL content.

[0029] S400 inputs the URL column, parsing flag column, and content source code column of each row of the table updated in step S300 into the map task function of the parsing job. The map function constructs output key-value pairs for each input row. For an input row, the value of the target URL column in that row is used as the key of the output key-value pair. If the content source code column of that row is empty, the value of the corresponding output key-value pair is "false", otherwise it is "true". If the content source code column of the input row is not empty and the parsing flag column is unparsed, the map function uses a customized regularization rule to parse the content source code of that row. The map function constructs an output key-value pair for each parsed URL, using the new URL as the key of the output key-value pair, and the value of the key-value pair is "false". The key-value pairs output by the map end are input into the combine function of the parsing job. The combine function iterates through the values ​​of all key-value pairs with the same URL as the key. If there is a "true" value, a key-value pair is output for that URL.<ur l,“true”> Otherwise, output a key-value pair for that URL.<ur l,“fa l se”> This step utilizes the combine function to significantly improve the efficiency of MapReduce jobs.

[0030] S500 inputs the output of the combine function described in step S400 into the reduce function of the parsing job. The operation logic of the reduce function is similar to that of the combine function. If there is a "true" value in the value list of key-value pairs with the same ur l as the key, then the parsing flag column of the corresponding row in the table updated in step S400 is set to parsed; otherwise, a new row is inserted in the table, the ur l column is the value of ur l, and the crawling and parsing flags are set to not crawled and not parsed.

[0031] S600 iteratively executes steps S300 to S500, incrementally crawling the target website data until the target data volume is reached or the target website is completely crawled.

[0032] Example:

[0033] This embodiment uses the crawling of data from 163AI.net as an example to provide a detailed description of the implementation of this method.

[0034] First, create a table in HBase containing three column families named "url", "flag", and "doc". Each row's "url" stores the URL of the data contained in that row; the "flag" column family contains two columns, "crawled" and "parsed", recording whether the content of the URL has been crawled and parsed, respectively; "doc" stores the source code of the URL. The hash value of the row's "url" is used as the row key. Initially, URLs such as "http: / / www.ai163.com / news / ", "http: / / www.ai163.com / vr / ", "http: / / www.ai163.com / ml / ", "http: / / www.ai163.com / 5g / ", "http: / / www.ai163.com / bigdata / ", "http: / / www.ai163.com / robot / ", "http: / / www.ai163.com / uav / ", "http: / / www.ai163.com / cloud / ", "http: / / www.ai163.com / iot / ", and "http: / / www.ai163.com / iov / " were collected as seed URLs. The hash values ​​of these URLs were calculated, and they were inserted into the table as new rows. The "url" column family contained the complete URLs, and both columns in the "flag" column family contained "fa lse". These websites have different prefixes and different parsing rules, so the seed URL and the corresponding parsing rules need to be saved in an XML file in advance. When crawling data, the rules can be obtained from the corresponding file to crawl smoothly. When it is necessary to add a seed URL for crawling, the value in the XML can be changed without changing the crawling logic code.

[0035] This embodiment designs two MapReduce jobs that are executed iteratively in a loop to incrementally crawl website data. The business logic of the map and reduce functions of the two jobs is described below:

[0036] (1) The crawling job will scan the entire table and transfer the values ​​of the “ur l” and “fl ag:crawl ered” columns of all rows to the map task.

[0037] The `map` function: During input data initialization, values ​​from two columns in the same row of the HBase table are grouped into a single key-value pair for processing. For each row, the `map` function checks if the value of the "flag:crawlered" column is "true". If so, it uses the HttpURLConnection library to crawl the URL's content. If the crawl is successful, the URL source code is saved in the "doc" column family of that row, named "document", and the value of the "flag:crawlered" column is set to "true". If the length of the crawled result is less than 100 characters, the crawl is considered to have failed, and no changes are made; the value of the "flag:crawlered" column remains "true".

[0038] The reduce function: The reduce function of the crawling job does not perform any extra processing; it is only used to support the map function in changing the corresponding values ​​in the HBase table.

[0039] (2) The parsing job transfers the values ​​of the “ur l”, “doc” and “fl ag:parsered” columns of all rows to the map task.

[0040] The `map` function, for each row of data, first checks if the "doc" column is empty. If empty, it outputs a key-value pair with the row's URL as the key and "false" as the value, indicating the row has not been parsed. Otherwise, it outputs a key-value pair with the row's URL as the key and "true" as the value, indicating the row has been parsed. Next, it checks if the "flag:parsed" column's value is "false". If so, it parses the content in the row's "doc:document" column. The parsing process involves using regularization rules to extract new URLs that meet certain conditions from the source code for incremental crawling. Since the selected website is a news blog focusing on artificial intelligence, the current rule uses the website's URL as the prefix. After parsing, the `map` function outputs a key-value pair with the newly parsed URL as the key and "false" as the value.

[0041] The `combine` function takes as input all key-value pairs with the same key (url) from the output of the `map` function of that node. Its processing logic is to iterate through all values ​​of the same url; if "true" exists among all values ​​for that url key, it means the content of that url has been parsed, and a key-value pair is output for that url.<ur l,”true”> Otherwise, if the URL content has not been parsed, a key-value pair will be output for the URL.<ur l,”fa l se”> .

[0042] The `reduce` function: The `combine` function is a function on the map side. The key-value pairs output by the map side serve as the input to the reduce side. Key-value pairs with the same key are processed by the same reduce function. The operation logic of the reduce function is similar to that of the `combine` function, except that: if a value of "true" exists in the value list of a certain URL, then the value of the "flag:parsed" column of the corresponding row is set to "true", indicating that the row has been parsed; otherwise, a new row is inserted, with the row key being the hash value of the URL, the "URL" column being the URL, and both columns of the "flag" column family being set to "true". The value of the "doc:document" column of this row will be obtained in the next iteration of the crawling job.

[0043] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0044] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A distributed data crawling method based on HBase and MapReduce, characterized in that, The method includes: S100 creates a table in HBase containing three column families: the target URL to be crawled, the crawling and parsing flags, and the source code of the content crawled from the target URL. S200 collects URLs with high out-degree from the target website as seed URLs and inserts them as new rows into the table described in step S100; S300 inputs the url column and crawl flag column of each row of the table described in step S200 into the map task function of the crawling job; the map task function of the crawling job determines that the crawl flag column of a certain row in the table is not crawled, and then uses the HttpURLConnection tool library to crawl the content source code of the url recorded in that row; if the crawling is successful, the crawled URL source code is saved in the content source code column family of that row in the table, and the crawl flag column of that row is set to crawled; if the length of the crawling result is less than 100 or the crawling fails, no changes are made; S400 inputs the URL column, parsing flag column, and content source code column of each row of the table updated in step S300 into the map task function of the parsing job; the map task function of the parsing job constructs output key-value pairs for each input row. For an input row, the value of the crawling target URL column in that row is used as the key of the output key-value pair. If the content source code column of that row is empty, the value of the corresponding output key-value pair is "false", otherwise it is "true"; if the content source code column of that input row is not empty and the parsing flag column is unparsed, the map function parses the content source code of that row using customized regularization rules; the key-value pairs output by the map task function of the parsing job are input into the combine function of the parsing job. S500 inputs the output of the combine function described in step S400 into the reduce function of the parsing job; S600 iteratively executes steps S300 to S500, incrementally crawling the target website data until the target data volume is reached or the target website is completely crawled.

2. The distributed data crawling method based on HBase and MapReduce according to claim 1, characterized in that, In step S100: The crawling and parsing flag column family contains two columns, which record the crawling flag and the parsing flag respectively. This table uses the hash value of each URL as the row key.

3. The distributed data crawling method based on HBase and MapReduce according to claim 2, characterized in that, In step S200: the crawling and parsing flags of the new row are set to not crawled and not parsed, respectively, and the content source code column is empty.

4. The distributed data crawling method based on HBase and MapReduce according to claim 1, characterized in that, The `combine` function iterates through all key-value pairs that use the same URL as their key. If any key-value pair with a given URL as its key contains the value "true", it outputs a key-value pair for that URL.<url, "true”> Otherwise, output a key-value pair for that URL.<url, "false”> .

5. The distributed data crawling method based on HBase and MapReduce according to claim 4, characterized in that, The operation logic of the reduce function in step S500 is as follows: if there is a "true" value in the list of key-value pairs with the same url as the key, then set the parsing flag column of the corresponding row in the table updated in step S400 to be parsed; otherwise, insert a new row in the table, with the url column containing the url value, and set the crawling and parsing flags to not crawled and not parsed.

Images