Web page data crawling method and device

Abstract

The present invention discloses a method and device for crawling web page data, relates to the technical field of data processing, improves the crawling efficiency of crawling data content in multiple web pages made by different web page templates, and the main technical scheme of the present invention is: classify and process multiple web pages to be crawled; crawl the source code file corresponding to each web page to be crawled by a general crawling module; use the exclusive content parsing module corresponding to each category to parse and process multiple source code files to obtain semi-structured data contained in each source code file; use the exclusive data processing module corresponding to each category to decompose and process multiple semi-structured data to obtain multiple structured data and multiple unstructured data contained in each semi-structured data; store multiple structured data and multiple unstructured data corresponding to each web page to be crawled. The present invention is applied to the process of crawling data content in multiple web pages made by different web page templates.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for crawling web page data. Background Art

[0002] With the increasing development of Internet technology, the era of big data has arrived. In the era of big data, the value of data is self-evident. For example, search engines, artificial intelligence, public opinion systems, etc. all need to obtain a large amount of data as the basis for research and development, making the role of web crawlers used to crawl data on the Internet increasingly important.

[0003] In the process of realizing the present invention, the inventor discovered the following technical problems in the prior art: because website staff use different web page templates when making web pages of different categories, when crawling data content in web pages made with different web page templates, it is necessary to use a web crawler corresponding to each web page template, and thus it is necessary to write a corresponding web crawler program for each web page template in advance, which results in low efficiency in crawling data content in multiple web pages made with different web page templates. Summary of the invention

[0004] In view of this, the present invention provides a method and device for crawling web page data, the main purpose of which is to improve the efficiency of crawling data content in multiple web pages made with different web page templates.

[0005] In order to solve the above problems, the present invention mainly provides the following technical solutions:

[0006] In one aspect, the present invention provides a method for crawling web page data, the method comprising:

[0007] Classifying the multiple web pages to be crawled so that the multiple web pages to be crawled included in each category are web pages made using the same web page template;

[0008] Crawl the source code file corresponding to each web page to be crawled by a general crawling module;

[0009] Using a dedicated content parsing module corresponding to each category to parse the multiple source code files contained in each category, so as to obtain semi-structured data contained in each source code file;

[0010] Decomposing the plurality of semi-structured data contained in each category using a dedicated data processing module corresponding to each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data;

[0011] The multiple structured data and the multiple unstructured data corresponding to each of the web pages to be crawled are stored.

[0012] Optionally, the classifying process of the plurality of web pages to be crawled includes:

[0013] receiving a classification instruction, and classifying the plurality of web pages to be crawled according to the classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs; or

[0014] A preset mapping relationship table is obtained, and a plurality of web pages to be crawled are classified and processed according to the preset mapping relationship table, wherein the preset mapping relationship table records a mapping relationship between a URL corresponding to each web page to be crawled and a corresponding category.

[0015] Optionally, after crawling the source code file corresponding to each of the web pages to be crawled by the general crawling module, the method further includes:

[0016] The exclusive content cleaning module corresponding to each category is used to clean the multiple source code files contained in each category, so as to remove useless data content in each source code file, regularize irregular data content in each source code file, and unify the data content structure of the multiple source code files contained in each category.

[0017] Optionally, after using the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data, the method further includes:

[0018] Extracting multiple attribute values ​​corresponding to each of the semi-structured data from the multiple structured data corresponding to each of the semi-structured data using a dedicated data processing module corresponding to each category;

[0019] Using a dedicated data processing module corresponding to each category to aggregate the multiple unstructured data corresponding to each of the semi-structured data, so as to obtain aggregated unstructured data corresponding to each of the semi-structured data;

[0020] The storing of the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled includes:

[0021] The multiple attribute values ​​and the aggregated unstructured data corresponding to each of the web pages to be crawled are stored.

[0022] Optionally, after storing the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled, the method further includes:

[0023] Determine whether the plurality of web pages to be crawled are all crawled successfully;

[0024] If there are any web pages that have not been crawled, the operation of crawling the multiple structured data and the multiple unstructured data corresponding to the web pages that have not been crawled is re-executed.

[0025] Optionally, before classifying the plurality of web pages to be crawled, the method further includes:

[0026] De-duplication processing is performed on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled.

[0027] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, an electronic device is provided, which includes: a memory, a processor, and a program stored in the memory and executable on the processor, and when the processor executes the program, the above-mentioned method for crawling web page data is implemented.

[0028] In order to achieve the above object, according to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed, the above-mentioned method for crawling web page data is implemented.

[0029] On the other hand, the present invention also provides a webpage data crawling device, the device comprising:

[0030] A classification unit, used for classifying a plurality of web pages to be crawled, so that the plurality of web pages to be crawled included in each category are web pages made using the same web page template;

[0031] A crawling unit, used for crawling the source code file corresponding to each web page to be crawled through a general crawling module;

[0032] A parsing unit, configured to parse the plurality of source code files contained in each category using a dedicated content parsing module corresponding to each category, so as to obtain semi-structured data contained in each source code file;

[0033] A decomposition unit, used for decomposing the plurality of semi-structured data contained in each category by using a dedicated data processing module corresponding to each category, so as to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data;

[0034] The storage unit is used to store multiple structured data and multiple unstructured data corresponding to each web page to be crawled.

[0035] Optionally, the classification unit includes:

[0036] A receiving module, configured to receive a classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs;

[0037] A first classification module, used for classifying the plurality of web pages to be crawled according to the classification instruction;

[0038] An acquisition module is used to acquire a preset mapping relationship table, wherein the preset mapping relationship table records the mapping relationship between the URL corresponding to each web page to be crawled and the category to which it belongs;

[0039] The second classification module is used to classify the multiple web pages to be crawled according to the preset mapping relationship table.

[0040] Optionally, the device further comprises:

[0041] A cleaning unit is used to use a dedicated content cleaning module corresponding to each category to clean the multiple source code files contained in each category after the crawling unit crawls the source code file corresponding to each of the to-be-crawled web pages through the general crawling module, so as to remove useless data content in each of the source code files, regularize irregular data content in each of the source code files, and unify the data content structure of the multiple source code files contained in each category.

[0042] Optionally, the device further comprises:

[0043] an extraction unit, configured to extract a plurality of attribute values ​​corresponding to each of the semi-structured data from the plurality of structured data corresponding to each of the semi-structured data using the exclusive data processing module corresponding to each category, after the decomposition unit uses the exclusive data processing module corresponding to each category to decompose the plurality of the semi-structured data contained in each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data;

[0044] a summarizing unit, configured to use a dedicated data processing module corresponding to each category to perform summarizing processing on a plurality of unstructured data corresponding to each of the semi-structured data, so as to obtain summarized unstructured data corresponding to each of the semi-structured data;

[0045] The storage unit is specifically used to store multiple attribute values ​​corresponding to each of the web pages to be crawled and the summarized unstructured data.

[0046] Optionally, the device further comprises:

[0047] A judging unit, configured to judge whether all the multiple web pages to be crawled are successfully crawled after the storage unit stores the multiple structured data and the multiple unstructured data corresponding to each of the web pages to be crawled;

[0048] The execution unit is used for re-executing the operation of crawling multiple structured data and multiple unstructured data corresponding to the uncrawled web pages when the judgment unit determines that there are uncrawled web pages.

[0049] Optionally, the device further comprises:

[0050] The deduplication unit is used to perform deduplication processing on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled before the classification unit classifies the multiple web pages to be crawled.

[0051] By means of the above technical solution, the technical solution provided by the present invention has at least the following advantages:

[0052] The present invention provides a web page data crawling method and device. Compared with the prior art in which a corresponding web crawler program is pre-written for each web page template, and then the web crawler programs corresponding to different web page templates are used to crawl the data content in web pages made with different web page templates, the present invention can, after receiving a crawling task containing URLs corresponding to multiple web pages to be crawled, firstly, classify the multiple web pages to be crawled based on the URLs corresponding to each web page to be crawled, so that the web pages to be crawled made with the same web page template are classified into the same category; then, crawl the source code file corresponding to each web page to be crawled by a general crawling module, and use a pre-written exclusive content parsing module corresponding to each category to parse the source code files corresponding to the multiple web pages to be crawled contained in each category, so as to extract the semi-structured data contained in the source code file corresponding to each web page to be crawled, and use a pre-written exclusive data processing module corresponding to each category to decompose the semi-structured data corresponding to the multiple web pages to be crawled contained in each category, so as to obtain multiple structured data and multiple unstructured data contained in each semi-structured data; finally, store the multiple structured data and multiple unstructured data corresponding to each web page to be crawled. Since there is no need to write a corresponding web crawler program for each category of web page template in advance, only a corresponding exclusive content parsing module and an exclusive data processing module need to be written in advance for each category of web page template, so that the data content in multiple web pages made with different web page templates can be crawled. Therefore, the crawling efficiency of crawling data content in multiple web pages made with different web page templates is improved.

[0053] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0055] Figure 1 A flow chart of a method for crawling web page data provided by an embodiment of the present invention;

[0056] Figure 2 A flow chart of another method for crawling web page data provided by an embodiment of the present invention;

[0057] Figure 3 A block diagram of a web page data crawling device provided by an embodiment of the present invention;

[0058] Figure 4 A block diagram of another web page data crawling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0060] The embodiment of the present invention provides a method for crawling web page data. Figure 1 As shown, the method includes:

[0061] 101. Categorize a plurality of web pages to be crawled so that the plurality of web pages to be crawled included in each category are web pages made using the same web page template.

[0062] In an embodiment of the present invention, after receiving a crawling task containing URLs (Uniform Resource Locator) corresponding to multiple web pages to be crawled, it is necessary to first classify the multiple web pages to be crawled based on the URLs corresponding to each web page to be crawled, so that the web pages to be crawled made using the same web page template are divided into the same category, and the web pages to be crawled made using different web page templates are divided into different categories.

[0063] 102. Crawl the source code file corresponding to each web page to be crawled through the general crawling module.

[0064] In an embodiment of the present invention, after a plurality of web pages to be crawled are classified and processed, the source code file corresponding to each web page to be crawled can be crawled through a general crawling module: the URL corresponding to each web page to be crawled is respectively input into the general crawling module, and the general crawling module can crawl the corresponding source code file based on the received URL, wherein the source code file corresponding to the web page to be crawled can specifically be: HTML file.

[0065] 103. Use the exclusive content parsing module corresponding to each category to parse multiple source code files contained in each category to obtain semi-structured data contained in each source code file.

[0066] It should be noted that, since the structure of the data content in multiple web pages to be crawled that are made using the same web page template is the same, a pre-written exclusive content parsing module corresponding to a certain category can be used to parse the source code files corresponding to the multiple web pages to be crawled contained in the category, thereby extracting the semi-structured data contained in each source code file, wherein the source code file corresponding to the web page to be crawled contains a code part and a data part (that is, semi-structured data).

[0067] In the embodiment of the present invention, since, in step 101, multiple web pages to be crawled have been classified and processed, and the web pages to be crawled made using the same web page template are classified into the same category, after the source code file corresponding to each web page to be crawled is obtained by crawling through the general crawling module, the pre-written exclusive content parsing module corresponding to each category can be used to parse the source code files corresponding to the multiple web pages to be crawled contained in each category, thereby extracting the semi-structured data contained in the source code file corresponding to each web page to be crawled. Specifically, in this step, the exclusive content parsing module parses the source code file corresponding to the web page to be crawled through the parsing engine.

[0068] 104. Use the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data.

[0069] In an embodiment of the present invention, after using an exclusive content parsing module corresponding to each category to parse and process the source code files corresponding to the multiple web pages to be crawled contained in each category, thereby obtaining the semi-structured data contained in the source code files corresponding to each web page to be crawled, a pre-written exclusive data processing module corresponding to each category can be used to decompose and process the semi-structured data corresponding to the multiple web pages to be crawled contained in each category (that is, decomposing the structured data and unstructured data contained in each semi-structured data), thereby obtaining multiple structured data and multiple unstructured data contained in each semi-structured data (that is, obtaining multiple structured data and multiple unstructured data corresponding to each web page to be crawled).

[0070] 105. Store multiple structured data and multiple unstructured data corresponding to each web page to be crawled.

[0071] In an embodiment of the present invention, after using an exclusive data processing module corresponding to each category to decompose and process the semi-structured data corresponding to the multiple web pages to be crawled contained in each category, thereby obtaining multiple structured data and multiple unstructured data corresponding to each web page to be crawled, the obtained multiple structured data and multiple unstructured data corresponding to each web page to be crawled can be stored.

[0072] Specifically, in an embodiment of the present invention, multiple structured data and multiple unstructured data corresponding to each web page to be crawled can be stored in a relational database, or multiple structured data and multiple unstructured data corresponding to each web page to be crawled can be stored in a non-relational database, or multiple structured data and multiple unstructured data corresponding to each web page to be crawled can be stored in a local Excel file, and the embodiment of the present invention does not specifically limit this.

[0073] The embodiment of the present invention provides a method for crawling web page data. Compared with the prior art in which a corresponding web crawler program is pre-written for each web page template, and then the web crawler programs corresponding to different web page templates are used to crawl the data content in web pages made with different web page templates, the embodiment of the present invention can, after receiving a crawling task including URLs corresponding to multiple web pages to be crawled, firstly, classify the multiple web pages to be crawled based on the URLs corresponding to each web page to be crawled, so that the web pages to be crawled made with the same web page template are classified into the same category; then, crawl the source code file corresponding to each web page to be crawled by a general crawling module, and use a pre-written exclusive content parsing module corresponding to each category to parse the source code files corresponding to the multiple web pages to be crawled contained in each category, so as to extract the semi-structured data contained in the source code file corresponding to each web page to be crawled, and use a pre-written exclusive data processing module corresponding to each category to decompose the semi-structured data corresponding to the multiple web pages to be crawled contained in each category, so as to obtain multiple structured data and multiple unstructured data contained in each semi-structured data; finally, store the multiple structured data and multiple unstructured data corresponding to each web page to be crawled. Since there is no need to write a corresponding web crawler program for each category of web page template in advance, only a corresponding exclusive content parsing module and an exclusive data processing module need to be written in advance for each category of web page template, so that the data content in multiple web pages made with different web page templates can be crawled. Therefore, the crawling efficiency of crawling data content in multiple web pages made with different web page templates is improved.

[0074] To explain in more detail below, an embodiment of the present invention provides another method for crawling web page data, specifically: Figure 2 As shown, the method includes:

[0075] 201. Categorize a plurality of web pages to be crawled so that the plurality of web pages to be crawled included in each category are web pages made using the same web page template.

[0076] In an embodiment of the present invention, after receiving a crawling task containing URLs corresponding to multiple web pages to be crawled, it is necessary to first classify the multiple web pages to be crawled based on the URLs corresponding to each web page to be crawled, so that the web pages to be crawled made using the same web page template are divided into the same category, and the web pages to be crawled made using different web page templates are divided into different categories.

[0077] Specifically, in this step, after receiving the classification instruction input by the staff and recording the category to which the URL corresponding to each web page to be crawled belongs, multiple web pages to be crawled can be classified according to the classification instruction (i.e., multiple web pages to be crawled can be classified according to the category to which the URL corresponding to each web page to be crawled belongs), so that the web pages to be crawled made using the same web page template are divided into the same category; or after obtaining the preset mapping relationship table recording the mapping relationship between the URL corresponding to each web page to be crawled and the category to which it belongs, multiple web pages to be crawled can be classified according to the preset mapping relationship table (i.e., multiple web pages to be crawled can be classified according to the mapping relationship between the URL corresponding to each web page to be crawled and the category to which it belongs), so that the web pages to be crawled made using the same web page template are divided into the same category. It should be noted that the preset mapping relationship table can be pre-written and stored by the staff according to the web page template used by each web page to be crawled, or it can be obtained and stored by the staff from a third-party platform, and the embodiment of the present invention does not specifically limit this.

[0078] Furthermore, in an embodiment of the present invention, in order to ensure that no repeated crawling work is performed, before the multiple web pages to be crawled are classified, the multiple web pages to be crawled can be deduplicated according to the URL corresponding to each web page to be crawled. It should be noted that in actual application, after the multiple web pages to be crawled are classified, the multiple web pages to be crawled can be deduplicated according to the URL corresponding to each web page to be crawled, and the embodiment of the present invention does not specifically limit this.

[0079] 202. Crawl the source code file corresponding to each web page to be crawled through the general crawling module.

[0080] Regarding step 202, crawling the source code file corresponding to each web page to be crawled by the general crawling module, please refer to Figure 1 The description of the corresponding parts of the embodiments of the present invention will not be repeated here.

[0081] Furthermore, in the embodiment of the present invention, the following three problems exist in the source code file corresponding to each web page to be crawled: (1) the source code file contains some useless data content. For example, the source code file corresponding to a web page to be crawled contains meaningless data content such as website LOGO, website logo, website statement, etc. (2) the source code file contains irregular data content. For example, the source code file corresponding to a web page to be crawled contains multiple meaningless blank lines. (3) the data content structure of the source code files corresponding to multiple web pages to be crawled made with the same web page template still has slight differences. For example, web page A to be crawled and web page B to be crawled belong to the same category. The source code file corresponding to web page A to be crawled records the size of painting a: Xcm*Ycm, while the source code file corresponding to web page B to be crawled records the size of painting b: height Mcm, width Ncm. Therefore, after the source code file corresponding to each web page to be crawled is obtained through the general crawling module, it is necessary to use the exclusive content cleaning module corresponding to each category to clean the source code files corresponding to the multiple web pages to be crawled contained in each category, so as to remove the useless data content in each source code file, regularize the irregular data content in each source code file, and unify the data content structure of the multiple source code files contained in each category, thereby solving the three problems mentioned above.

[0082] 203. Use the dedicated content parsing module corresponding to each category to parse the multiple source code files included in each category to obtain semi-structured data included in each source code file.

[0083] Regarding step 203, using the dedicated content parsing module corresponding to each category to parse multiple source code files contained in each category to obtain the semi-structured data contained in each source code file, please refer to Figure 1 The description of the corresponding parts of the embodiments of the present invention will not be repeated here.

[0084] 204. Use the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data.

[0085] Regarding step 204, using the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data, please refer to Figure 1 The description of the corresponding parts of the embodiments of the present invention will not be repeated here.

[0086] Furthermore, in an embodiment of the present invention, after using an exclusive data processing module corresponding to each category to decompose and process the semi-structured data corresponding to the multiple web pages to be crawled contained in each category, thereby obtaining multiple structured data and multiple unstructured data contained in each semi-structured data, in order to make the data content corresponding to each web page to be crawled obtained more concise, the exclusive data processing module corresponding to each category can also be used to extract multiple attribute values ​​contained in each semi-structured data from the multiple structured data corresponding to each semi-structured data contained in each category (that is, extract and obtain multiple attribute values ​​corresponding to each web page to be crawled), and use the exclusive data processing module corresponding to each category to summarize and process the multiple unstructured data corresponding to each semi-structured data contained in each category to obtain the summarized unstructured data corresponding to each semi-structured data (that is, summarize and obtain the summarized unstructured data corresponding to each web page to be crawled), and then the multiple attribute values ​​and summarized unstructured data corresponding to each web page to be crawled can be stored, wherein each structured data consists of its corresponding name and the attribute value corresponding to the name.

[0087] 205. Store multiple structured data and multiple unstructured data corresponding to each web page to be crawled.

[0088] Regarding step 205, storing multiple structured data and multiple unstructured data corresponding to each web page to be crawled, please refer to Figure 1 The description of the corresponding parts of the embodiments of the present invention will not be repeated here.

[0089] 206. Determine whether multiple web pages to be crawled are all crawled successfully.

[0090] In an embodiment of the present invention, after storing multiple structured data and multiple unstructured data corresponding to each web page to be crawled, it is also necessary to determine whether multiple web pages to be crawled are all crawled successfully, so that when it is determined that there are web pages that have not been successfully crawled (uncrawled web pages), the data content in the uncrawled web pages can be re-crawled.

[0091] 207. If there are any web pages that have not been crawled, re-execute the operation of crawling multiple structured data and multiple unstructured data corresponding to the web pages that have not been crawled.

[0092] In an embodiment of the present invention, when it is determined that there are uncrawled web pages, it is necessary to re-execute the operation of crawling multiple structured data and multiple unstructured data corresponding to the uncrawled web pages, that is, adopt the method described in the above steps 202-205 to re-crawl the multiple structured data and multiple unstructured data corresponding to the uncrawled web pages, thereby ensuring that the data content in each web page to be crawled is successfully crawled.

[0093] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, an embodiment of the present invention further provides an electronic device, which includes: a memory, a processor, and a program stored in the memory and executable on the processor, and when the processor executes the program, the above-mentioned method for crawling web page data is implemented.

[0094] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed, the above-mentioned method for crawling web page data is implemented.

[0095] Furthermore, as a response to the above Figure 1 , Figure 2 The implementation of the method shown in the figure, the embodiment of the present invention provides a device for crawling web page data. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the present device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can implement all the contents of the aforementioned method embodiment. The device is used to improve the crawling efficiency of data content in multiple web pages made by different web page templates, specifically as follows Figure 3 As shown, the device comprises:

[0096] The classification unit 301 is used to classify the multiple web pages to be crawled so that the multiple web pages to be crawled included in each category are web pages made using the same web page template;

[0097] A crawling unit 302 is used to crawl the source code file corresponding to each web page to be crawled through a general crawling module;

[0098] A parsing unit 303, configured to parse the plurality of source code files contained in each category using a dedicated content parsing module corresponding to each category, so as to obtain semi-structured data contained in each source code file;

[0099] A decomposition unit 304 is used to decompose the multiple semi-structured data contained in each category using a dedicated data processing module corresponding to each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data;

[0100] The storage unit 305 is used to store multiple structured data and multiple unstructured data corresponding to each of the web pages to be crawled.

[0101] Further, such as Figure 4 As shown, the classification unit 301 includes:

[0102] The receiving module 3011 is used to receive a classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs;

[0103] A first classification module 3012, used for classifying the plurality of web pages to be crawled according to the classification instruction;

[0104] The acquisition module 3013 is used to acquire a preset mapping relationship table, wherein the preset mapping relationship table records the mapping relationship between the URL corresponding to each web page to be crawled and the category to which it belongs;

[0105] The second classification module 3014 is used to classify the multiple web pages to be crawled according to the preset mapping relationship table.

[0106] Further, such as Figure 4 As shown, the device also includes:

[0107] The cleaning unit 306 is used to use the exclusive content cleaning module corresponding to each category to clean the multiple source code files contained in each category after the crawling unit 302 crawls the source code file corresponding to each of the to-be-crawled web pages through the general crawling module, so as to remove useless data content in each of the source code files, regularize the irregular data content in each of the source code files, and unify the data content structure of the multiple source code files contained in each category.

[0108] Further, such as Figure 4 As shown, the device also includes:

[0109] The extraction unit 307 is used to extract multiple attribute values ​​corresponding to each of the semi-structured data from the multiple structured data corresponding to each of the semi-structured data using the exclusive data processing module corresponding to each category after the decomposition unit 304 uses the exclusive data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each of the semi-structured data;

[0110] A summarizing unit 308 is used to use a dedicated data processing module corresponding to each category to perform summarizing processing on the multiple unstructured data corresponding to each of the semi-structured data to obtain the summarized unstructured data corresponding to each of the semi-structured data;

[0111] The storage unit 305 is specifically used to store multiple attribute values ​​corresponding to each of the web pages to be crawled and the summarized unstructured data.

[0112] Further, such as Figure 4 As shown, the device also includes:

[0113] A judging unit 309 is used to judge whether all the web pages to be crawled are successfully crawled after the storage unit 305 stores the multiple structured data and the multiple unstructured data corresponding to each web page to be crawled;

[0114] The execution unit 310 is used for re-executing the operation of crawling multiple structured data and multiple unstructured data corresponding to the uncrawled web pages when the judgment unit 309 judges that there are uncrawled web pages.

[0115] Further, such as Figure 4 As shown, the device also includes:

[0116] The deduplication unit 311 is used to perform deduplication processing on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled before the classification unit 301 classifies the multiple web pages to be crawled.

[0117] In summary, the embodiment of the present invention provides a method and device for crawling web page data. Compared with the prior art in which a corresponding web crawler program is pre-written for each web page template, and then the web crawler programs corresponding to different web page templates are used to crawl the data content in web pages made with different web page templates, the embodiment of the present invention can, after receiving a crawling task including URLs corresponding to multiple web pages to be crawled, firstly, classify the multiple web pages to be crawled based on the URLs corresponding to each web page to be crawled, so that the web pages to be crawled made with the same web page template are classified into the same category; then, the universal crawling module crawls the corresponding URLs of each web page to be crawled. The method comprises the following steps: a source code file is prepared, and a pre-written exclusive content parsing module corresponding to each category is used to parse the source code files corresponding to the multiple web pages to be crawled contained in each category, thereby extracting the semi-structured data contained in the source code files corresponding to each web page to be crawled, and using a pre-written exclusive data processing module corresponding to each category to decompose the semi-structured data corresponding to the multiple web pages to be crawled contained in each category, thereby obtaining multiple structured data and multiple unstructured data contained in each semi-structured data; finally, the multiple structured data and multiple unstructured data corresponding to each web page to be crawled are stored. Since there is no need to pre-write a corresponding web crawler program for each category of web page template, only the corresponding exclusive content parsing module and exclusive data processing module need to be pre-written for each category of web page template, so that the data content in multiple web pages made with different web page templates can be crawled. Therefore, the crawling efficiency of crawling the data content in multiple web pages made with different web page templates is improved.

[0118] The web page data crawling device includes a processor and a memory. The above-mentioned classification unit, crawling unit, parsing unit, decomposition unit and storage unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0119] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the crawling efficiency of crawling data content in multiple web pages made by different web page templates can be improved by adjusting kernel parameters.

[0120] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0121] An embodiment of the present invention provides an electronic device, which includes: a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the web page data crawling method is implemented.

[0122] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed, the method for crawling web page data described above is implemented.

[0123] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, the following steps are implemented:

[0124] Classifying the multiple web pages to be crawled so that the multiple web pages to be crawled included in each category are web pages made using the same web page template;

[0125] Crawl the source code file corresponding to each web page to be crawled by a general crawling module;

[0126] Using a dedicated content parsing module corresponding to each category to parse the multiple source code files contained in each category, so as to obtain semi-structured data contained in each source code file;

[0127] Decomposing the plurality of semi-structured data contained in each category using a dedicated data processing module corresponding to each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data;

[0128] The multiple structured data and the multiple unstructured data corresponding to each of the web pages to be crawled are stored.

[0129] Furthermore, the classification process of the plurality of web pages to be crawled includes:

[0130] receiving a classification instruction, and classifying the plurality of web pages to be crawled according to the classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs; or

[0131] A preset mapping relationship table is obtained, and a plurality of web pages to be crawled are classified and processed according to the preset mapping relationship table, wherein the preset mapping relationship table records a mapping relationship between a URL corresponding to each web page to be crawled and a corresponding category.

[0132] Furthermore, after crawling the source code file corresponding to each of the web pages to be crawled by the general crawling module, the method further includes:

[0133] The exclusive content cleaning module corresponding to each category is used to clean the multiple source code files contained in each category, so as to remove useless data content in each source code file, regularize irregular data content in each source code file, and unify the data content structure of the multiple source code files contained in each category.

[0134] Furthermore, after using the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data, the method further includes:

[0135] Extracting multiple attribute values ​​corresponding to each of the semi-structured data from the multiple structured data corresponding to each of the semi-structured data using a dedicated data processing module corresponding to each category;

[0136] Using a dedicated data processing module corresponding to each category to aggregate the multiple unstructured data corresponding to each of the semi-structured data, so as to obtain aggregated unstructured data corresponding to each of the semi-structured data;

[0137] The storing of the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled includes:

[0138] The multiple attribute values ​​and the aggregated unstructured data corresponding to each of the web pages to be crawled are stored.

[0139] Furthermore, after storing the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled, the method further includes:

[0140] Determine whether the plurality of web pages to be crawled are all crawled successfully;

[0141] If there are any web pages that have not been crawled, the operation of crawling the multiple structured data and the multiple unstructured data corresponding to the web pages that have not been crawled is re-executed.

[0142] Furthermore, before classifying the plurality of web pages to be crawled, the method further includes:

[0143] De-duplication processing is performed on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled.

[0144] The device in this article can be a server, PC, etc.

[0145] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program code that initializes the following method steps: classifying a plurality of web pages to be crawled so that the plurality of web pages to be crawled contained in each category are web pages made using the same web page template; crawling the source code file corresponding to each of the web pages to be crawled by a general crawling module; parsing the plurality of source code files contained in each category using an exclusive content parsing module corresponding to each category to obtain semi-structured data contained in each of the source code files; decomposing the plurality of semi-structured data contained in each category using an exclusive data processing module corresponding to each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data; and storing the plurality of structured data and a plurality of unstructured data corresponding to each of the web pages to be crawled.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (traHsitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0155] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for crawling web page data, characterized in that: The method comprises: Classifying the multiple web pages to be crawled so that the multiple web pages to be crawled included in each category are web pages made using the same web page template, wherein the structures of the data contents in the multiple web pages to be crawled made using the same web page template are the same; Crawl the source code file corresponding to each web page to be crawled by a general crawling module; Using a dedicated content parsing module corresponding to each category to parse the multiple source code files contained in each category, so as to obtain semi-structured data contained in each source code file; Decomposing the plurality of semi-structured data contained in each category using a dedicated data processing module corresponding to each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data; Storing a plurality of structured data and a plurality of unstructured data corresponding to each of the web pages to be crawled; The classifying process of the plurality of web pages to be crawled includes: receiving a classification instruction, and classifying the plurality of web pages to be crawled according to the classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs; or A preset mapping relationship table is obtained, and a plurality of web pages to be crawled are classified according to the preset mapping relationship table, wherein the preset mapping relationship table records a mapping relationship between a URL corresponding to each web page to be crawled and a corresponding category.

2. The method according to claim 1, characterized in that After crawling the source code file corresponding to each of the web pages to be crawled by the general crawling module, the method further includes: The exclusive content cleaning module corresponding to each category is used to clean the multiple source code files contained in each category, so as to remove useless data content in each source code file, regularize irregular data content in each source code file, and unify the data content structure of the multiple source code files contained in each category.

3. The method according to claim 1, characterized in that After using the dedicated data processing module corresponding to each category to decompose the multiple semi-structured data contained in each category to obtain multiple structured data and multiple unstructured data contained in each semi-structured data, the method further includes: Extracting multiple attribute values ​​corresponding to each of the semi-structured data from the multiple structured data corresponding to each of the semi-structured data using a dedicated data processing module corresponding to each category; Using a dedicated data processing module corresponding to each category to aggregate the multiple unstructured data corresponding to each of the semi-structured data, so as to obtain aggregated unstructured data corresponding to each of the semi-structured data; The storing of the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled includes: The multiple attribute values ​​and the aggregated unstructured data corresponding to each of the web pages to be crawled are stored.

4. The method according to claim 1, characterized in that After storing the plurality of structured data and the plurality of unstructured data corresponding to each of the web pages to be crawled, the method further includes: Determine whether the plurality of web pages to be crawled are all crawled successfully; If there are any web pages that have not been crawled, the operation of crawling the multiple structured data and the multiple unstructured data corresponding to the web pages that have not been crawled is re-executed.

5. The method according to any one of claims 1 to 4, characterized in that Before classifying the plurality of web pages to be crawled, the method further includes: De-duplication processing is performed on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled.

6. A web page data crawling device, characterized in that: The device comprises: A classification unit, used for classifying a plurality of web pages to be crawled, so that the plurality of web pages to be crawled contained in each category are web pages made using the same web page template, wherein the structure of the data content in the plurality of web pages to be crawled made using the same web page template is the same; A crawling unit, used for crawling the source code file corresponding to each web page to be crawled through a general crawling module; A parsing unit, configured to parse the plurality of source code files contained in each category using a dedicated content parsing module corresponding to each category, so as to obtain semi-structured data contained in each source code file; A decomposition unit, used to decompose the plurality of semi-structured data contained in each category using a dedicated data processing module corresponding to each category, so as to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data; A storage unit, used for storing a plurality of structured data and a plurality of unstructured data corresponding to each of the web pages to be crawled; The taxonomic units include: A receiving module, configured to receive a classification instruction, wherein the classification instruction records the category to which the uniform resource locator URL corresponding to each web page to be crawled belongs; A first classification module, used for classifying the plurality of web pages to be crawled according to the classification instruction; An acquisition module is used to acquire a preset mapping relationship table, wherein the preset mapping relationship table records the mapping relationship between the URL corresponding to each web page to be crawled and the category to which it belongs; The second classification module is used to classify the multiple web pages to be crawled according to the preset mapping relationship table.

7. The device according to claim 6, characterized in that The device also includes: A cleaning unit is used to use a dedicated content cleaning module corresponding to each category to clean the multiple source code files contained in each category after the crawling unit crawls the source code file corresponding to each of the to-be-crawled web pages through the general crawling module, so as to remove useless data content in each of the source code files, regularize irregular data content in each of the source code files, and unify the data content structure of the multiple source code files contained in each category.

8. The device according to claim 6, characterized in that The device also includes: an extraction unit, configured to extract a plurality of attribute values ​​corresponding to each of the semi-structured data from the plurality of structured data corresponding to each of the semi-structured data using the exclusive data processing module corresponding to each category, after the decomposition unit uses the exclusive data processing module corresponding to each category to decompose the plurality of the semi-structured data contained in each category to obtain a plurality of structured data and a plurality of unstructured data contained in each of the semi-structured data; a summarizing unit, configured to use a dedicated data processing module corresponding to each category to perform summarizing processing on a plurality of unstructured data corresponding to each of the semi-structured data, so as to obtain summarized unstructured data corresponding to each of the semi-structured data; The storage unit is specifically used to store multiple attribute values ​​corresponding to each of the web pages to be crawled and the summarized unstructured data.

9. The device according to claim 6, characterized in that The device also includes: A judging unit, configured to judge whether all the multiple web pages to be crawled are successfully crawled after the storage unit stores the multiple structured data and the multiple unstructured data corresponding to each of the web pages to be crawled; The execution unit is used for re-executing the operation of crawling multiple structured data and multiple unstructured data corresponding to the uncrawled web pages when the judgment unit determines that there are uncrawled web pages.

10. The device according to any one of claims 6 to 9, characterized in that: The device also includes: The deduplication unit is used to perform deduplication processing on the multiple web pages to be crawled according to the URL corresponding to each of the web pages to be crawled before the classification unit classifies the multiple web pages to be crawled.

11. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for crawling web page data as described in any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is running, the method for crawling web page data as described in any one of claims 1 to 5 is implemented.

Images