Web page data capture method, device, storage medium and electronic device

Abstract

The present invention discloses a method, device, storage medium and electronic device for crawling web page data. The method comprises: obtaining candidate web page links that match web page link information in a pre-configured web page link information set from web page links on a target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; searching for a target web page whose web page type is a target type from the candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in the pre-configured web page link information set; crawling web page data from the target web page according to the crawling rule. The present invention solves the technical problem of low crawling efficiency when crawling web page data in the related art.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device, storage medium and electronic device for capturing web page data. Background Art

[0002] In order to understand the development of the industry more quickly and comprehensively, it is sometimes necessary to crawl the content of some websites. The current crawling method is to start from one link and continuously expand new links and then crawl, and then expand new links, over and over again. However, this method has many problems. For example, the crawling is relatively blind, and a lot of useless content may be captured. The site resources may not be crawled completely, many links are not crawled, and the crawling is incomplete, and it is difficult to judge whether the crawling is complete.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, storage medium and electronic device for capturing web page data, so as to at least solve the technical problem of low capture efficiency when capturing web page data in the related art.

[0005] According to one aspect of an embodiment of the present invention, a method for crawling web page data is provided, comprising: acquiring candidate web page links that match web page link information in a preconfigured web page link information set from web page links on a target website, wherein the web page links on the target website include: web page links corresponding to a homepage of the target website, and web page links corresponding to each level of web pages under the homepage; searching for a target web page whose web page type is a target type from candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in a preconfigured web page link information set; and crawling web page data from the target web page according to the crawling rule.

[0006] According to another aspect of an embodiment of the present invention, a device for crawling web page data is also provided, including: an acquisition module, used to acquire candidate web page links that match the web page link information in a pre-configured web page link information set from the web page links on the target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; a search module, used to search for a target web page with a target type of web page type among the candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in the pre-configured web page link information set; and a crawling module, used to crawl web page data from the target web page according to the crawling rule.

[0007] According to another aspect of an embodiment of the present invention, a storage medium is provided, wherein a computer program is stored in the storage medium, wherein the computer program is configured to execute any of the methods described above when running.

[0008] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute any of the above methods through the computer program.

[0009] In an embodiment of the present invention, a candidate web page link matching the web page link information in a pre-configured web page link information set is obtained from web page links on a target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; a target web page of a target type is searched for in the candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in the pre-configured web page link information set; a web page link information set is pre-configured in a manner of crawling web page data from the target web page according to the crawling rule, a candidate web page link matching the web page link information in the pre-configured web page link information set is obtained from web page links on the target website, and then a target web page configured with the crawling rule is searched in the candidate web page links, and web page data is crawled from the target web page, so that the target web page can be screened out from a large number of web pages on the target website to crawl web page data, thereby avoiding blind crawling of web page data, improving the validity of the crawled web page data, and thus achieving the technical effect of improving the crawling efficiency when crawling web page data, thereby solving the technical problem of low crawling efficiency when crawling web page data in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a schematic diagram of an optional method for crawling web page data according to an embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of an application environment of an optional webpage data crawling method according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of an optional method for crawling web page data according to an optional implementation mode of the present invention;

[0014] Figure 4is a schematic diagram of another optional method for crawling web page data according to an optional implementation mode of the present invention;

[0015] Figure 5 is a schematic diagram of an optional webpage data crawling device according to an embodiment of the present invention;

[0016] Figure 6 is a schematic diagram of an application scenario of an optional method for crawling web page data according to an embodiment of the present invention; and

[0017] Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] According to one aspect of an embodiment of the present invention, a method for crawling web page data is provided. Figure 1 As shown, the method includes:

[0021] S102, acquiring candidate web page links that match web page link information in a pre-configured web page link information set from web page links on the target website, wherein the web page links on the target website include: web page links corresponding to a homepage of the target website, and web page links corresponding to each level of web pages under the homepage;

[0022] S104, searching for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page link, wherein the target type is used to indicate that a crawling rule for the target web page is configured in a pre-configured web page link information set;

[0023] S106, crawling web page data from the target web page according to the crawling rules.

[0024] Optionally, in this embodiment, the webpage data crawling method can be applied to Figure 2 In the hardware environment constituted by the server 202 shown in FIG. Figure 2 As shown, the server 202 obtains candidate web page links that match the web page link information in a pre-configured web page link information set from the web page links on the target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; searches for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in the pre-configured web page link information set; and crawls web page data from the target web page according to the crawling rule.

[0025] Optionally, in this embodiment, the method for capturing web page data can be applied to, but not limited to, the scenario of capturing web page data on a target website. The target website can be, but not limited to, various types of websites, such as online education websites, instant messaging websites, community space websites, game websites, shopping websites, browser websites, financial websites, multimedia websites, live broadcast websites, etc. Specifically, it can be applied to, but not limited to, the scenario of capturing web page data on a target website on the multimedia website, or it can be applied to, but not limited to, the scenario of capturing web page data on a target website on the browser website, so as to improve the efficiency of capturing web page data. The above is only an example, and this embodiment does not make any limitation to this.

[0026] Optionally, in this embodiment, the form of the web page link information in the pre-configured web page link information set may include, but is not limited to: a URL of a web page link, a regular expression, and the like.

[0027] Optionally, in this embodiment, the web links on the target website include: the web links corresponding to the homepage of the target website, and the web links corresponding to each level of web pages under the homepage. For example, the target website may be, but is not limited to, a video website, and the web links corresponding to its homepage may be www.host.com. The first-level links under the homepage may include tv.host.com, movie.host.com, dongman.host.com, etc. The links under these links, which may also be called the second-level links under the homepage, may include: the links under tv.host.com: lishi.tv.host.com, guzhuang.tv.host.com, shizhuang.tv.host.com, etc., the links under movie.host.com: dongzuo.movie.host.com, aiqing.movie.host.com, j ingsong.movie.host.com, etc., the next level links of dongman.host.com: rexue.dongman.host.com, xiaoyuan.dongman.host.com, lishi.dongman.host.com, etc., the next level links of these links, also known as the third level links under the homepage, may include: tv1.tv.host.com, tv2.tv.host.com, etc. under lishi.tv.host.com, tv3.tv.host.com, tv4.tv.host.com, etc. under guzhuang.tv.host.com, the third level web page links may also have fourth level web page links, and so on.

[0028] It should be noted that the above embodiment is only an example of the hierarchical form of web page links. Other forms of web page links are similar to this and will not be described in detail here.

[0029] Optionally, in this embodiment, a crawling rule is configured for the target page in the pre-configured web link information set, then the web page that can be found in the pre-configured web link information set according to the crawling rule is the target web page, and web page data is crawled from the target web page according to the pre-configured crawling rule, thereby ensuring the validity of the acquired web page data and avoiding crawling too much useless content. For example: the above tv3.tv.host.com is the play web page of TV series 3, and tv4.tv.host.com is the play web page of TV series 4. The crawling rules configured for them may all be: crawling the TV series name, crawling the TV series episode number, crawling the TV series introduction content, etc.

[0030] In an optional embodiment, if Figure 3 As shown, the web page link information in the pre-configured web page link information set includes: home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15, wherein the pre-configured web page link information set also configures the crawling rules of web page 7 and web page 11 as rule 1, rule 2, and rule 3, and the web page links on the target website include: home page 1, web page 1, web page 2, ..., and web page 48. The home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15 are obtained from the home page 1, web page 1, web page 2, ..., and web page 48 as candidate web page links that match the web page link information in the pre-configured web page link information set, and then the target web pages configured with the crawling rules are searched in the candidate web pages pointed to by the candidate web page links (home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15), and web page data is crawled from web page 7 and web page 11 according to rule 1, rule 2, and rule 3.

[0031] It can be seen that through the above steps, a web page link information set is pre-configured, and candidate web page links that match the web page link information in the pre-configured web page link information set are obtained from the web page links on the target website, and then the target web page configured with the crawling rules is searched in the candidate web page links, and web page data is crawled from the target web page. The target web page can be screened out from a large number of web pages on the target website to crawl web page data, thereby avoiding blind crawling of web page data and improving the effectiveness of the crawled web page data, thereby achieving the technical effect of improving the efficiency of crawling when crawling web page data, and then solving the technical problem of low crawling efficiency when crawling web page data in related technologies.

[0032] As an optional solution, obtaining candidate web page links that match web page link information in a pre-configured web page link information set from web page links on a target website includes: crawling web page links from a to-be-processed web page on the target website, wherein the to-be-processed web page is a web page pointed to by web page link information for which crawling rules are not configured in the pre-configured web page link information set; searching for candidate web page links that match web page link information in the pre-configured web page link information set from the crawled web page links;

[0033] Searching for a target webpage with a target webpage type among the candidate webpages pointed to by the candidate webpage link includes: performing the following steps on each candidate webpage: obtaining the webpage type of the candidate webpage; and confirming the candidate webpage as the target webpage when the webpage type of the candidate webpage is the target type.

[0034] Optionally, in this embodiment, the web pages pointed to by the web page link information for which no crawling rules are configured in the pre-configured web page link information set can be used as web pages to be processed. These web pages to be processed can be used to expand web page links, crawl web page links from the web pages to be processed, match the crawled web page links with the web page link information in the pre-configured web page link information set, find out the successfully matched web page links as candidate web page links, and then obtain the candidate web pages pointed to by the candidate web page links, and after obtaining the web page type of each candidate web page, determine the candidate web page with the target type as the target web page.

[0035] As an optional solution, after obtaining the webpage type of the candidate webpage, the following is further included:

[0036] S1, when the web page type of the candidate web page is not the target type, the web page to be processed is updated to the candidate web page, and the process jumps to execute the steps of crawling web page links from the web page to be processed in the target website and searching for candidate web page links that match the web page link information in the pre-configured web page link information set in the crawled web page links.

[0037] Optionally, in this embodiment, for a candidate web page whose web page type is not the target type, it is treated as a new web page to be processed to perform the steps of crawling web page links and matching web page links.

[0038] Optionally, in this embodiment, the candidate web page links can be obtained step by step starting from the homepage of the target website, but not limited to. Figure 4 As shown, the pre-configured web page link information set configures the web page link information in the form of a directory tree. In the directory tree, the web page link information that is not configured with crawling rules can be called index pages, and the web page link information that is configured with crawling rules can be called result pages. The root site is the homepage of the target website. The web page links of the next level of the root site are configured with web page link information of three index pages, index page 1, index page 2, and index page 3. The web page links of the next level of index page 1 are configured with web page link information of three index pages, index page 4, index page 5, and index page 6. Index page 4 has result page 1, and result page 1 is configured with crawling rule 1. Index page 5 has result page 2, and result page 2 is configured with crawling rule 2. Index page 6 has result page 3, and result page 3 is configured with crawling rule 3. Index page 2 has result page 4, and result page 4 is configured with crawling rule 4. Index page 3 has result page 5, and result page 5 is configured with crawling rule 5.

[0039] First, take the homepage as the web page to be processed, crawl web links from the homepage, and search for candidate web links that match the web link information in the pre-configured web link information set among the crawled web links, so that index page 1, index page 2, and index page 3 among the web links crawled on the homepage are matched, and then obtain the web page types of index page 1, index page 2, and index page 3 respectively. Since the web page types of index page 1, index page 2, and index page 3 are not the target types, the web pages to be processed are updated to index page 1, index page 2, and index page 3 respectively, and jump to execute the steps of crawling web links and matching web link information respectively.

[0040] Taking index page 1 as an example, web page links are captured from index page 1, and candidate web page links that match the web page link information in the pre-configured web page link information set are searched among the captured web page links. It can be found that index page 4, index page 5, and index page 6 are matched among the web page links captured on index page 1, and then the web page types of index page 4, index page 5, and index page 6 are obtained respectively. Since the web page types of index page 4, index page 5, and index page 6 are not the target types, the web pages to be processed are updated to index page 4, index page 5, and index page 6 respectively, and the steps of capturing web page links and matching web page link information are jumped to execute respectively.

[0041] Taking index page 4 as an example, web page links are captured from index page 4, and candidate web page links matching web page link information in the pre-configured web page link information set are searched in the captured web page links. It can be found that result page 1 is matched in the web page links captured on index page 4, and then the web page type of result page 1 is obtained. Since the web page type of result page 1 is the target type, result page 1 is confirmed as the target web page. Finally, the target web pages can be found to be result page 1, result page 2, result page 3, result page 4 and result page 5, and web page data is captured according to their corresponding crawling rules.

[0042] As an optional solution, obtaining candidate web page links that match web page link information in a pre-configured web page link information set from web page links on the target website includes:

[0043] S1, obtaining one or more regular expressions from a pre-configured web page link information set, wherein the web page link information includes one or more regular expressions;

[0044] S2, matching the web links on the target website with one or more regular expressions;

[0045] S3: Determine, among the web page links on the target website, a web page link that successfully matches at least one of the one or more regular expressions as a candidate web page link.

[0046] Optionally, in this embodiment, the web page link information configured in the pre-configured web page link information set may be, but is not limited to, a regular expression. For example, the regular expression configured for the web page links lishi.tv.host.com, guzhuang.tv.host.com, and shizhuang.tv.host.com may be, but is not limited to, *.tv.host.com, and the regular expression configured for the web page links dongzuo.movie.host.com, aiqing.movie.host.com, and jingsong.movie.host.com may be, but is not limited to, *.movie.host.com, and so on.

[0047] As an optional solution, matching the web links on the target website with one or more regular expressions includes:

[0048] S1, obtaining the uniform resource locator of the web page link on the target website;

[0049] S2, matches the uniform resource locator with one or more regular expressions.

[0050] Optionally, in this embodiment, the webpage link and the webpage link information are matched by, but not limited to, matching the uniform resource locator (URL) of the webpage link with a regular expression. For example, the following uniform resource locators of the webpage links are captured on the webpage tv.host.com: guanggao1.host.com, guanggao2.host.comlishi.tv.host.com, lishi.tv.host.com, guzhuang.tv.host.com, shizhuang.tv.host.com, then the candidate webpage links obtained by matching these uniform resource locators with the regular expression *.tv.host.com are lishi.tv.host.com, guzhuang.tv.host.com, and shizhuang.tv.host.com.

[0051] As an optional solution, searching for a target web page whose web page type is the target type among the candidate web pages pointed to by the candidate web page link includes:

[0052] S1, searching for a crawling rule corresponding to a candidate web page pointed to by a candidate web page link from a pre-configured web page link information set;

[0053] S2: determining the candidate web page pointed to by the candidate web page link of the crawling rule found from the pre-configured web page link information set as a target web page of the target type.

[0054] Optionally, in this embodiment, in order to enhance the pertinence and purposefulness of the webpage content crawling process, webpage link information of webpage links is pre-configured, and webpages of different purposes are divided into different types. Some webpages are used for extending links, which are called index pages. These index pages only configure the webpage links to be crawled in the pre-configured webpage link information set, but do not configure crawling rules. In other words, only webpage links are crawled in these index pages, but no specific webpage data is crawled. Some webpages are used for crawling data, which are called result pages. These result pages not only configure the webpage links to be crawled, but also configure crawling rules in the pre-configured webpage link information set, so as to obtain the required webpage data from these webpages.

[0055] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0056] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0057] According to another aspect of the embodiment of the present invention, a web page data capture device for implementing the web page data capture method is also provided. Figure 5 As shown, the device comprises:

[0058] The acquisition module 52 is used to acquire candidate web page links that match the web page link information in the pre-configured web page link information set from the web page links on the target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage;

[0059] A search module 54 is used to search for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page link, wherein the target type is used to indicate that a crawling rule for the target web page is configured in a pre-configured web page link information set;

[0060] The crawling module 56 is used to crawl web page data from the target web page according to the crawling rules.

[0061] Optionally, in this embodiment, the webpage data grabbing device can be applied to Figure 2 In the hardware environment constituted by the server 202 shown in FIG. Figure 2 As shown, the server 202 obtains candidate web page links that match the web page link information in a pre-configured web page link information set from the web page links on the target website, wherein the web page links on the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; searches for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page links, wherein the target type is used to indicate that a crawling rule for the target web page is configured in the pre-configured web page link information set; and crawls web page data from the target web page according to the crawling rule.

[0062] Optionally, in this embodiment, the web page data capture device can be, but is not limited to, applied to the scenario of capturing web page data on a target website. The target website can be, but is not limited to, various types of websites, such as online education websites, instant messaging websites, community space websites, game websites, shopping websites, browser websites, financial websites, multimedia websites, live broadcast websites, etc. Specifically, it can be, but is not limited to, applied to the scenario of capturing web page data on a target website on the multimedia website, or it can also be, but is not limited to, applied to the scenario of capturing web page data on a target website on the browser website, so as to improve the efficiency of capturing web page data. The above is only an example, and this embodiment does not make any limitation to this.

[0063] Optionally, in this embodiment, the form of the web page link information in the pre-configured web page link information set may include, but is not limited to: a URL of a web page link, a regular expression, and the like.

[0064] Optionally, in this embodiment, the web links on the target website include: the web links corresponding to the homepage of the target website, and the web links corresponding to each level of web pages under the homepage. For example, the target website may be, but is not limited to, a video website, and the web links corresponding to its homepage may be www.host.com. The first-level links under the homepage may include tv.host.com, movie.host.com, dongman.host.com, etc. The links under these links, which may also be called the second-level links under the homepage, may include: the links under tv.host.com: lishi.tv.host.com, guzhuang.tv.host.com, shizhuang.tv.host.com, etc., the links under movie.host.com: dongzuo.movie.host.com, aiqing.movie.host.com, j ingsong.movie.host.com, etc., the next level links of dongman.host.com: rexue.dongman.host.com, xiaoyuan.dongman.host.com, lishi.dongman.host.com, etc., the next level links of these links, also known as the third level links under the homepage, may include: tv1.tv.host.com, tv2.tv.host.com, etc. under lishi.tv.host.com, tv3.tv.host.com, tv4.tv.host.com, etc. under guzhuang.tv.host.com, the third level web page links may also have fourth level web page links, and so on.

[0065] It should be noted that the above embodiment is only an example of the hierarchical form of web page links. Other forms of web page links are similar to this and will not be described in detail here.

[0066] Optionally, in this embodiment, a crawling rule is configured for the target page in the pre-configured web link information set, then the web page that can be found in the pre-configured web link information set according to the crawling rule is the target web page, and web page data is crawled from the target web page according to the pre-configured crawling rule, thereby ensuring the validity of the acquired web page data and avoiding crawling too much useless content. For example: the above tv3.tv.host.com is the play web page of TV series 3, and tv4.tv.host.com is the play web page of TV series 4. The crawling rules configured for them may all be: crawling the TV series name, crawling the TV series episode number, crawling the TV series introduction content, etc.

[0067] In an optional embodiment, if Figure 3 As shown, the web page link information in the pre-configured web page link information set includes: home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15, wherein the pre-configured web page link information set also configures the crawling rules of web page 7 and web page 11 as rule 1, rule 2, and rule 3, and the web page links on the target website include: home page 1, web page 1, web page 2, ..., and web page 48. The home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15 are obtained from the home page 1, web page 1, web page 2, ..., and web page 48 as candidate web page links that match the web page link information in the pre-configured web page link information set, and then the target web pages configured with the crawling rules are searched in the candidate web pages pointed to by the candidate web page links (home page 1, web page 2, web page 3, web page 5, web page 7, web page 11, and web page 15), and web page data is crawled from web page 7 and web page 11 according to rule 1, rule 2, and rule 3.

[0068] It can be seen that through the above-mentioned device, a web page link information set is pre-configured, and candidate web page links that match the web page link information in the pre-configured web page link information set are obtained from the web page links on the target website, and then the target web page configured with the crawling rules is searched in the candidate web page links, and the web page data is crawled from the target web page. The target web page can be screened out from a large number of web pages on the target website to crawl the web page data, thereby avoiding blind crawling of web page data and improving the effectiveness of the crawled web page data, thereby achieving the technical effect of improving the efficiency of crawling when crawling web page data, and then solving the technical problem of low crawling efficiency when crawling web page data in the related technology.

[0069] As an optional solution, the acquisition module is used to: crawl web page links from a to-be-processed web page in a target website, wherein the to-be-processed web page is a web page pointed to by web page link information for which no crawling rules are configured in a pre-configured web page link information set; search for candidate web page links that match web page link information in the pre-configured web page link information set in the crawled web page links;

[0070] The search module is used to perform the following steps on each candidate web page: obtaining the web page type of the candidate web page; and confirming the candidate web page as the target web page when the web page type of the candidate web page is the target type.

[0071] Optionally, in this embodiment, the web pages pointed to by the web page link information for which no crawling rules are configured in the pre-configured web page link information set can be used as web pages to be processed. These web pages to be processed can be used to expand web page links, crawl web page links from the web pages to be processed, match the crawled web page links with the web page link information in the pre-configured web page link information set, find out the successfully matched web page links as candidate web page links, and then obtain the candidate web pages pointed to by the candidate web page links, and after obtaining the web page type of each candidate web page, determine the candidate web page with the target type as the target web page.

[0072] As an optional solution, the above device also includes:

[0073] A processing module is used to update the web page to be processed to a candidate web page when the web page type of the candidate web page is not the target type, and jump to execute the steps of crawling web page links from the web page to be processed in the target website and searching for candidate web page links that match the web page link information in a pre-configured web page link information set in the crawled web page links.

[0074] Optionally, in this embodiment, for a candidate web page whose web page type is not the target type, it is treated as a new web page to be processed to perform the steps of crawling web page links and matching web page links.

[0075] Optionally, in this embodiment, the candidate web page links can be obtained step by step starting from the homepage of the target website, but not limited to. Figure 4 As shown, the pre-configured web page link information set configures the web page link information in the form of a directory tree. In the directory tree, the web page link information that is not configured with crawling rules can be called index pages, and the web page link information that is configured with crawling rules can be called result pages. The root site is the homepage of the target website. The web page links of the next level of the root site are configured with web page link information of three index pages, index page 1, index page 2, and index page 3. The web page links of the next level of index page 1 are configured with web page link information of three index pages, index page 4, index page 5, and index page 6. Index page 4 has result page 1, and result page 1 is configured with crawling rule 1. Index page 5 has result page 2, and result page 2 is configured with crawling rule 2. Index page 6 has result page 3, and result page 3 is configured with crawling rule 3. Index page 2 has result page 4, and result page 4 is configured with crawling rule 4. Index page 3 has result page 5, and result page 5 is configured with crawling rule 5.

[0076] First, take the homepage as the web page to be processed, crawl web links from the homepage, and search for candidate web links that match the web link information in the pre-configured web link information set among the crawled web links, so that index page 1, index page 2, and index page 3 among the web links crawled on the homepage are matched, and then obtain the web page types of index page 1, index page 2, and index page 3 respectively. Since the web page types of index page 1, index page 2, and index page 3 are not the target types, the web pages to be processed are updated to index page 1, index page 2, and index page 3 respectively, and jump to execute the steps of crawling web links and matching web link information respectively.

[0077] Taking index page 1 as an example, web page links are captured from index page 1, and candidate web page links that match the web page link information in the pre-configured web page link information set are searched among the captured web page links. It can be found that index page 4, index page 5, and index page 6 are matched among the web page links captured on index page 1, and then the web page types of index page 4, index page 5, and index page 6 are obtained respectively. Since the web page types of index page 4, index page 5, and index page 6 are not the target types, the web pages to be processed are updated to index page 4, index page 5, and index page 6 respectively, and the steps of capturing web page links and matching web page link information are jumped to execute respectively.

[0078] Taking index page 4 as an example, web page links are captured from index page 4, and candidate web page links matching web page link information in the pre-configured web page link information set are searched in the captured web page links. It can be found that result page 1 is matched in the web page links captured on index page 4, and then the web page type of result page 1 is obtained. Since the web page type of result page 1 is the target type, result page 1 is confirmed as the target web page. Finally, the target web pages can be found to be result page 1, result page 2, result page 3, result page 4 and result page 5, and web page data is captured according to their corresponding crawling rules.

[0079] As an optional solution, the acquisition module includes:

[0080] An acquisition unit, configured to acquire one or more regular expressions from a pre-configured web page link information set, wherein the web page link information includes one or more regular expressions;

[0081] A matching unit, used to match web page links on a target website with one or more regular expressions;

[0082] The first determining unit is used to determine, among the web page links on the target website, a web page link that successfully matches at least one regular expression among the one or more regular expressions as a candidate web page link.

[0083] Optionally, in this embodiment, the web page link information configured in the pre-configured web page link information set may be, but is not limited to, a regular expression. For example, the regular expression configured for the web page links lishi.tv.host.com, guzhuang.tv.host.com, and shizhuang.tv.host.com may be, but is not limited to, *.tv.host.com, and the regular expression configured for the web page links dongzuo.movie.host.com, aiqing.movie.host.com, and jingsong.movie.host.com may be, but is not limited to, *.movie.host.com, and so on.

[0084] As an optional solution, the matching unit includes:

[0085] An acquisition subunit is used to acquire the uniform resource locator of the web page link on the target website;

[0086] The matching subunit is used to match the URL with one or more regular expressions.

[0087] Optionally, in this embodiment, the webpage link and the webpage link information are matched by, but not limited to, matching the uniform resource locator (URL) of the webpage link with a regular expression. For example, the following uniform resource locators of the webpage links are captured on the webpage tv.host.com: guanggao1.host.com, guanggao2.host.comlishi.tv.host.com, lishi.tv.host.com, guzhuang.tv.host.com, shizhuang.tv.host.com, then the candidate webpage links obtained by matching these uniform resource locators with the regular expression *.tv.host.com are lishi.tv.host.com, guzhuang.tv.host.com, and shizhuang.tv.host.com.

[0088] As an optional solution, the search module includes:

[0089] A search unit, used to search for a crawling rule corresponding to a candidate web page pointed to by a candidate web page link from a pre-configured web page link information set;

[0090] The second searching unit is used to determine the candidate web page pointed to by the candidate web page link of the crawling rule found in the pre-configured web page link information set as a target web page of the target type.

[0091] Optionally, in this embodiment, in order to enhance the pertinence and purposefulness of the webpage content crawling process, webpage link information of webpage links is pre-configured, and webpages of different purposes are divided into different types. Some webpages are used for extending links, which are called index pages. These index pages only configure the webpage links to be crawled in the pre-configured webpage link information set, but do not configure crawling rules. In other words, only webpage links are crawled in these index pages, but no specific webpage data is crawled. Some webpages are used for crawling data, which are called result pages. These result pages not only configure the webpage links to be crawled, but also configure crawling rules in the pre-configured webpage link information set, so as to obtain the required webpage data from these webpages.

[0092] The application environment of the embodiment of the present invention can be but not limited to the application environment in the above embodiment, which will not be described in detail in this embodiment. The embodiment of the present invention provides an optional specific application example for implementing the above real-time communication connection method.

[0093] As an optional embodiment, the webpage data crawling method described above can be applied to, but is not limited to, Figure 6 In the scenario of crawling web page data of a video website as shown in . In this scenario, the URLs of the website are first aggregated to build a URL directory tree, and then the URLs are graded into "index pages" and "result pages", where the index page is responsible for expanding links and the result page is responsible for crawling web page data. Figure 6 The following is a directory tree of a video website. All child nodes are "result pages", which are pages that need to capture web page data. Index pages can be expanded to "index pages" or "result pages". The hierarchy from the root node to the result page is uncertain. The capture method is as follows: Horizontal expansion can be used to schedule the capture of related pages, that is, to capture web page links or web page data by layer. Expand links, promote links, and capture web page links on the index page. A timeout operation can be set. If the links on the previous layer are not captured within the specified time, the next layer can be moved to. When processing the "result page", regular expressions can be configured to capture the desired content based on the characteristics of each URL and the content that needs to be captured.

[0094] In this embodiment, the homepage of the video website is the index page, which is expanded to "index of tv", "index of moive", "index of zy", and "index of tv" is expanded to index pages such as "costume drama", "martial arts", and "history". "Costume drama" can be expanded to the result page of "A Step into the Past".

[0095] According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the above-mentioned web page data crawling, such as Figure 7As shown, the electronic device includes: one or more (only one is shown in the figure) processors 702, a memory 704, a sensor 706, an encoder 708 and a transmission device 710, the memory stores a computer program, and the processor is configured to execute the steps in any of the above method embodiments through the computer program.

[0096] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0097] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0098] S1, obtaining candidate web page links that match web page link information in a pre-configured web page link information set from web page links on a target website, wherein the web page links on the target website include: web page links corresponding to a homepage of the target website, and web page links corresponding to each level of web pages under the homepage;

[0099] S2, searching for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page link, wherein the target type is used to indicate that a crawling rule for the target web page is configured in a pre-configured web page link information set;

[0100] S3, crawls web page data from the target web page according to the crawling rules.

[0101] Alternatively, a person skilled in the art may understand that: Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 7 The structure of the electronic device is not limited. Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 7 Different configurations shown.

[0102] Among them, the memory 702 can be used to store software programs and modules, such as program instructions / modules corresponding to the webpage data capture method and device in the embodiment of the present invention. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, to realize the control method of the above-mentioned target component. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include a memory remotely arranged relative to the processor 704, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The transmission device 710 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 710 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 710 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0104] Specifically, the memory 702 is used to store application programs.

[0105] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0106] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0107] S1, obtaining candidate web page links that match web page link information in a pre-configured web page link information set from web page links on a target website, wherein the web page links on the target website include: web page links corresponding to a homepage of the target website, and web page links corresponding to each level of web pages under the homepage;

[0108] S2, searching for a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page link, wherein the target type is used to indicate that a crawling rule for the target web page is configured in a pre-configured web page link information set;

[0109] S3, crawls web page data from the target web page according to the crawling rules.

[0110] Optionally, the storage medium is also configured to store a computer program for executing the steps included in the method in the above embodiment, which will not be described in detail in this embodiment.

[0111] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0112] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0113] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0114] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for crawling web page data, characterized in that: include: The web page links of the index page in the web page set included in the target website are used as candidate web page links, wherein the index page is a web page for which no web page data crawling rules are configured, and the web page links included in the target website include: web page links corresponding to the homepage of the target website, and web page links corresponding to each level of web pages under the homepage; In the case where the next-level link included in the current index page points to the index page of the next level, taking the webpage link of the index page of the next level as the candidate webpage link associated with the current index page; In the case where the next-level link included in the current index page points to the next-level result page, using the web page link of the next-level result page as the target web page link associated with the current index page, wherein the result page is a web page configured with a web page data crawling rule; Acquire the candidate web page links that match the web page link information in the web page link information set from the web page links on the target website; Searching for a target web page whose web page type is the target type among the candidate web pages pointed to by the candidate web page link; The webpage data is crawled from the target webpage according to the crawling rules.

2. The method according to claim 1, characterized in that Acquiring candidate web links that match web link information in a pre-configured web link information set from web links on a target website includes: crawling web links from a to-be-processed web page in the target website, wherein the to-be-processed web page is a web page pointed to by web link information that is not configured with the crawling rule in the pre-configured web link information set; searching for the candidate web links that match web link information in the pre-configured web link information set from the crawled web links; Searching for a target webpage whose webpage type is the target type among the candidate webpages pointed to by the candidate webpage links includes: performing the following steps on each of the candidate webpages: obtaining the webpage type of the candidate webpage; and confirming the candidate webpage as the target webpage if the webpage type of the candidate webpage is the target type.

3. The method according to claim 2, characterized in that After obtaining the webpage type of the candidate webpage, the method further includes: In the case that the web page type of the candidate web page is not the target type, the web page to be processed is updated to the candidate web page, and the process jumps to execute the steps of crawling web page links from the web page to be processed in the target website and searching for the candidate web page links that match the web page link information in the pre-configured web page link information set in the crawled web page links.

4. The method according to claim 1, characterized in that Acquiring candidate webpage links that match the webpage link information in the webpage link information set from the webpage links on the target website includes: Acquire one or more regular expressions from a preconfigured web page link information set, wherein the web page link information includes the one or more regular expressions; Matching the web page links on the target website with the one or more regular expressions; A web page link on the target website that successfully matches at least one regular expression among the one or more regular expressions is determined as the candidate web page link.

5. The method according to claim 4, characterized in that Matching the webpage link on the target website with the one or more regular expressions includes: Obtaining the uniform resource locator of the web page link on the target website; The uniform resource locator is matched against the one or more regular expressions.

6. The method according to claim 1, characterized in that Searching for a target web page of a target type among the candidate web pages pointed to by the candidate web page link includes: Searching for a crawling rule corresponding to the candidate web page pointed to by the candidate web page link from a pre-configured web page link information set; The candidate web page pointed to by the candidate web page link of the crawling rule found in the pre-configured web page link information set is determined as the target web page whose web page type is the target type.

7. A webpage data crawling device, characterized in that: include: An acquisition module is used to use a web page link of an index page in a web page set included in a target website as a candidate web page link, wherein the index page is a web page for which a web page data crawling rule is not configured, and the web page links included in the target website include: a web page link corresponding to a homepage of the target website, and a web page link corresponding to each level of web pages under the homepage; in the case where a next-level link included in a current index page points to the index page of the next level, use the web page link of the index page of the next level as the candidate web page link associated with the current index page; in the case where a next-level link included in the current index page points to a result page of the next level, use the web page link of the result page of the next level as the target web page link associated with the current index page, wherein the result page is a web page for which a web page data crawling rule is configured; and obtain, from the web page links on the target website, the candidate web page link that matches the web page link information in the web page link information set; A search module, used for searching a target web page whose web page type is a target type among the candidate web pages pointed to by the candidate web page link; The crawling module is used to crawl web page data from the target web page according to the crawling rules.

8. The device according to claim 7, characterized in that The acquisition module is used to: crawl webpage links from the webpage to be processed in the target website, wherein the webpage to be processed is a webpage pointed to by webpage link information in a pre-configured webpage link information set that is not configured with the crawling rule; search the crawled webpage links for the candidate webpage links that match the webpage link information in the pre-configured webpage link information set; The search module is used to perform the following steps on each candidate web page: obtaining the web page type of the candidate web page; and confirming the candidate web page as the target web page when the web page type of the candidate web page is the target type.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images