Crawler capturing method and device, electronic equipment and storage medium

By automatically adjusting crawler seeds and requests through configuration information and preset target depth, the problems of low crawler crawling efficiency and complex process in the prior art are solved, and more efficient information acquisition and simplified logical processing are achieved.

CN118897909APending Publication Date: 2024-11-05ZHEJIANG ZEEKR INTELLIGENT TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310507075.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-04
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The existing crawler crawling methods are not efficient, the process is cumbersome, and require a lot of manual coding and complex logical processing.

Method used

By obtaining the configuration information of the task to be crawled, determine the crawl depth of the current crawler result, and automatically adjust the crawler seeds and requests according to the preset target depth requirements until the target depth is reached, simplifying the crawler logic and reducing the amount of code.

Benefits of technology

It improves the efficiency of crawler crawling, reduces manual workload, is more flexible in logic, simplifies the process, and achieves more efficient information acquisition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897909A_ABST
    Figure CN118897909A_ABST
Patent Text Reader

Abstract

The invention provides a crawler capturing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining configuration information of a to-be-crawled task, a current crawler seed and a current crawler request; according to the current crawler seed and the current crawler request, executing a to-be-crawled task to obtain a current crawler capture result; determining the current capture depth of the current crawler capture result according to the configuration information; if the current capture depth does not meet the preset target capture depth requirement, obtaining a next crawler seed and a next crawler request according to the current crawler capture result; and taking the next crawler seed and the next crawler request as a current crawler seed and a current crawler request, executing a step of executing the to-be-crawled task according to the current crawler seed and the current crawler request to obtain a current crawler capture result, and obtaining a target crawler capture result until the current capture depth meets a preset target capture depth requirement. According to the method, the crawler capturing efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of web crawlers, and in particular to a crawler capture method, device, electronic device and storage medium. Background Art

[0002] The era of Internet big data provides people with a large amount of information sources, which brings convenience to life. However, the emergence of massive data also increases the threshold and difficulty of information screening. For example, when a user enters a keyword in a search engine, the returned results may include a lot of other content that is irrelevant to the content the user wants to obtain. Therefore, a way to efficiently obtain effective information is needed.

[0003] At present, in the prior art, effective information can be captured by crawlers. When performing crawler capture, technicians usually perform crawler coding according to the task to be crawled. After the crawler coding is completed, the task to be crawled is executed to obtain crawler capture results containing effective information.

[0004] However, the existing crawler crawling methods have the problems of being cumbersome and inefficient. Summary of the invention

[0005] The embodiments of the present application provide a crawler crawling method, device, electronic device and storage medium to solve the problems of the existing crawler crawling method being complicated and inefficient.

[0006] In a first aspect, an embodiment of the present application provides a crawler crawling method, which is applied to a crawler crawling system, including:

[0007] Get the configuration information of the crawler task to be crawled, the current crawler seed and the current crawler request;

[0008] According to the current crawler seed and the current crawler request, execute the pending crawler task to obtain the current crawler crawling result;

[0009] According to the configuration information, determine the current crawling depth of the current crawler crawling results;

[0010] If the current crawling depth does not meet the preset target crawling depth requirement, the next crawler seed and the next crawler request are obtained according to the current crawling result of the crawler, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed;

[0011] The next crawler seed and the next crawler request are used as the current crawler seed and the current crawler request, and the steps of executing the pending crawler task and obtaining the current crawler crawling result are executed according to the current crawler seed and the current crawler request, until the current crawling depth meets the preset target crawling depth requirement, and the target crawler crawling result is obtained.

[0012] In an embodiment of the present application, obtaining configuration information of the crawler task to be performed, the current crawler seed, and the current crawler request includes:

[0013] In response to the user's configuration operation, obtain the configuration information of the crawler task to be processed;

[0014] Determine the crawler seed for the crawler task to be completed based on the configuration information;

[0015] Get the crawler request based on the crawler seed and configuration information.

[0016] In an embodiment of the present application, in response to a user's configuration operation, obtaining configuration information of a crawler task to be performed includes:

[0017] In response to the user's template generation operation, a configuration template corresponding to the crawler task to be performed is obtained, where the configuration template includes a plurality of configuration units, and the configuration units are units selected by the user from a preset configuration database according to the crawler task to be performed;

[0018] In response to the user's write operation on the configuration unit according to the task to be crawled, the configuration information of the task to be crawled is obtained.

[0019] In an embodiment of the present application, determining the crawler seed of the crawler task to be crawled according to the configuration information includes:

[0020] Determine task scheduling information based on configuration information;

[0021] According to the task scheduling information, determine the execution priority of the crawler task to be executed and the target crawler node for executing the crawler task to be executed, wherein the target crawler node is a node among the crawler nodes that meets the load requirements, and the crawler node is a node in the crawler crawling system used to execute the crawler task;

[0022] After the target crawler node receives the crawler execution instruction, the execution type of the crawler execution instruction is determined according to the crawler execution instruction, and the execution type includes a new task type and a breakpoint resume task type;

[0023] According to the execution type, the crawler seed of the task to be crawled is obtained.

[0024] In the embodiment of the present application, when the execution type is a new task type, the crawler seed of the task to be crawled is obtained according to the execution type, including:

[0025] Determine, according to the execution type, iteration information in the configuration information and an iteration type of the iteration information;

[0026] According to the iteration type, a link list of tasks to be crawled is obtained, wherein the link list includes an initial crawler seed;

[0027] The initial crawler seed is marked to obtain the crawler seed of the crawler task to be completed.

[0028] In the embodiment of the present application, when the execution type is a breakpoint resume task type, a crawler seed for the task to be crawled is generated according to the execution type, including:

[0029] According to the execution type, the crawler seed is read in the crawler task to be crawled, and the crawler seed of the crawler task to be crawled is obtained.

[0030] In an embodiment of the present application, according to the configuration information, determining the current crawling depth of the current crawler crawling result includes:

[0031] According to the configuration information, determine the current iteration crawling depth of the crawler task to be crawled;

[0032] According to the current iteration crawl depth, determine the current crawl depth of the current crawler crawl results.

[0033] In the embodiment of the present application, after determining the current crawling depth of the current crawler crawling result according to the configuration information, the method further includes:

[0034] If the current crawling depth meets the preset target crawling depth requirement, all crawling results of the crawler executing the crawler task are obtained;

[0035] Determine the target crawler crawling results based on all crawler crawling results.

[0036] In the embodiment of the present application, if the current crawling depth does not meet the preset target crawling depth requirement, then after obtaining the next crawler seed and the next crawler request according to the current crawler crawling result, the method further includes:

[0037] Get the crawler seed to be executed of the crawler task to be executed in the crawler node;

[0038] Compare the seed of the crawler to be executed with the seed of the next crawler to obtain a comparison result;

[0039] If the comparison result indicates that the crawler seed to be executed and the next crawler seed do not match, then executing the step of obtaining the next crawler request according to the configuration information and the next crawler seed;

[0040] If the comparison result indicates that the crawler seed to be executed matches the next crawler seed, the next crawler seed is deleted, and the crawler seed to be executed is used as the next crawler seed, and the step of obtaining the next crawler request according to the configuration information and the next crawler seed is executed.

[0041] In the embodiment of the present application, after executing the pending crawler task according to the current crawler seed and the current crawler request and obtaining the current crawler crawling result, the method further includes:

[0042] Verify the current crawler crawling results and obtain the verification results;

[0043] If the verification result indicates that the current crawler crawling result is normal, the current crawler crawling result is saved;

[0044] If the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed.

[0045] In an embodiment of the present application, if the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result includes:

[0046] If the verification result indicates that the current crawler crawling result is abnormal, the number of retries for the current crawler crawling result is obtained;

[0047] If the number of retries does not meet the preset retry number threshold requirement, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed;

[0048] If the number of retries meets the preset retry number threshold requirement, it is determined that the status of the current crawler crawling result is abnormal, and the execution of the pending crawler task is terminated.

[0049] In a second aspect, an embodiment of the present application provides a crawler grasping device, comprising:

[0050] The acquisition module is used to obtain the configuration information of the crawler task to be processed, the current crawler seed, and the current crawler request;

[0051] The execution module is used to execute the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result;

[0052] A determination module, used to determine the current crawling depth of the current crawler crawling result according to the configuration information;

[0053] The first obtaining module is used to obtain the next crawler seed and the next crawler request according to the current crawler crawling result if the current crawling depth does not meet the preset target crawling depth requirement, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed;

[0054] The second obtaining module is used to use the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, execute the pending crawler task according to the current crawler seed and the current crawler request, and obtain the steps of the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and obtains the target crawler crawling result.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0056] Memory stores computer-executable instructions;

[0057] The processor executes the computer-executable instructions stored in the memory to implement the crawler crawling method of the embodiment of the present application.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the crawler crawling method of the embodiment of the present application.

[0059] The crawler crawling method, device, electronic device and storage medium provided by the present application obtain configuration information of the task to be crawled, the current crawler seed and the current crawler request; execute the task to be crawled according to the current crawler seed and the current crawler request to obtain the current crawler crawling result; determine the current crawling depth of the current crawler crawling result according to the configuration information; if the current crawling depth does not meet the preset target crawling depth requirement, obtain the next crawler seed and the next crawler request according to the current crawler crawling result, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed; use the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, and execute the crawling result according to the current crawler seed. The sub-task and the current crawler request execute the task to be crawled, and obtain the steps of the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and then obtains the target crawler crawling result. The configuration information of the task to be crawled can be used to determine the current crawling depth of the current crawler result, and based on the comparison between the current crawling depth and the preset target crawling depth requirement, it can quickly determine whether the task to be crawled is completed, so as to crawl and crawl according to the next crawler seed and the next crawler request. In this way, it gets rid of the fixed process in the traditional method, reduces the workload of the staff, makes the logic of the crawler more flexible, reduces the amount of code, simplifies the process, and achieves the effect of improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A schematic diagram of the process of crawling the crawler provided in the embodiment of the present application;

[0062] Figure 2 A flowchart of another crawler crawling method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of the module flow in the crawler system provided in an embodiment of the present application;

[0064] Figure 4 A flow chart of the scheduler operation logic provided in the embodiment of the present application;

[0065] Figure 5 A flow chart of the grabber logic provided in the embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of a crawler grabbing device provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0068] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0070] In the prior art, when crawling, workers are usually required to encode the crawler. When the crawled page has multiple layers of information, workers are required to encode the pages of different layers separately, which makes the entire encoding workload larger, the process more complicated, and the efficiency of the entire crawling is also lower.

[0071] In order to solve the above problems, the crawler crawling method provided by the present application can determine the current crawling depth of the current crawler result by configuring the configuration information of the task to be crawled, and compare the current crawling depth with the preset target crawling depth requirement to quickly determine whether the task to be crawled is completed, so as to perform crawler crawling according to the next crawler seed and the next crawler request. In this way, it gets rid of the fixed process method in the traditional method, reduces the workload of the staff, makes the logic of the crawler more flexible, reduces the amount of code, simplifies the process, and solves the problem of low efficiency of the entire crawler process.

[0072] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0073] The execution subject of the crawler crawling method provided in the embodiment of the present application can be a server. Among them, the server can be a mobile phone, a tablet, a computer and other devices. This embodiment does not impose any special restrictions on the implementation method of the execution subject, as long as the execution subject can obtain the configuration information of the crawler task to be crawled, the current crawler seed and the current crawler request; according to the current crawler seed and the current crawler request, the crawler task to be crawled is executed to obtain the current crawler crawling result; according to the configuration information, the current crawling depth of the current crawler crawling result is determined; if the current crawling depth does not meet the preset target crawling depth requirement, then according to the current crawler crawling result, the next crawler seed and the next crawler request are obtained, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed; the next crawler seed and the next crawler request are used as the current crawler seed and the current crawler request, and the steps of executing the crawler task to be crawled according to the current crawler seed and the current crawler request to obtain the current crawler crawling result are executed until the current crawling depth meets the preset target crawling depth requirement, and the target crawler crawling result is obtained.

[0074] A web crawler refers to a program that automatically extracts web pages from the World Wide Web. Traditional crawlers start with the URLs (Uniform Resource Locators) of one or more initial web pages, obtain the URLs on the initial web pages, and in the process of crawling web pages, continuously extract new URLs from the current page and put them into a queue until a certain stop condition of the system is met.

[0075] Crawler requests (requests): is an http request library based on native python http encapsulation.

[0076] Scrapy is a fast, high-level web crawler framework written for Python. It can be used to obtain web sites and extract structured data from pages.

[0077] Figure 1 The flowchart of the crawler crawling method provided in the embodiment of the present application is shown in FIG. The execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 1 As shown, the method may include:

[0078] S101, obtaining configuration information of the crawler task to be performed, the current crawler seed and the current crawler request.

[0079] The task to be crawled may refer to a task to be crawled and captured. For example, the task to be crawled may be a crawler task of obtaining relevant information about “target news” from a “news website”.

[0080] The configuration information may include the configuration information of the crawler entry and the configuration information of the task configuration, wherein the configuration information of the crawler entry may include: entry naming, target crawling URL, request type, iterative crawling depth, positioning details list xpath rule, depth-maintained list positioning xpath rule, saved page URL regularization, etc. The configuration information of the task configuration may include: task name, task priority, task scheduling cycle configuration, and designated running node.

[0081] The current crawler seed may refer to the crawler seed obtained according to the target crawl URL in the configuration information during the current iteration of crawling.

[0082] The current crawler request may refer to a crawler request obtained according to the current crawler seed.

[0083] Among them, in the embodiment of the present application, the method of obtaining the configuration information of the crawler task to be crawled, the current crawler seed and the current crawler request may include:

[0084] In response to the user's configuration operation, obtain the configuration information of the crawler task to be processed;

[0085] Determine the crawler seed for the crawler task to be completed based on the configuration information;

[0086] Get the crawler request based on the crawler seed and configuration information.

[0087] Among them, in the embodiment of the present application, in response to the user's configuration operation, the method for obtaining the configuration information of the crawler task to be performed may include:

[0088] In response to the user's template generation operation, a configuration template corresponding to the crawler task to be performed is obtained, where the configuration template includes a plurality of configuration units, and the configuration units are units selected by the user from a preset configuration database according to the crawler task to be performed;

[0089] In response to the user's write operation on the configuration unit according to the task to be crawled, the configuration information of the task to be crawled is obtained.

[0090] The template generation operation may refer to the process of generating a configuration template according to the configuration information of the crawler task. The configuration template may refer to the framework content in the process of coding the crawler task. The configuration template may be composed of multiple configuration units, each of which may be a framework template for items to be filled in. For example, for a framework template with an entrance named "entrance_name" in the configuration information, after calling the configuration unit, the user does not need to write the code for the configuration unit, thereby improving efficiency.

[0091] The preset configuration database may refer to a database from which the user retrieves configuration units when configuring a template. The configuration database may store a variety of different configuration units, so that when the user needs to write crawler code, the configuration unit that the user needs to use can be extracted from the configuration database.

[0092] The write operation may refer to the process of a user filling in the content of the configuration unit according to the crawler task to be crawled. For example, when the crawler task to be crawled is a crawling task for a homepage address of "http: / / china.xxxxxxx.com.cn", the configuration unit may be "target crawling URL", and the write operation may refer to the process of writing "http: / / china.xxxxxxx.com.cn" into "target crawling URL" to obtain "http: / / china.xxxxxxx.com.cn / 5bd5639ca3101a87ca8ff636 / page_1.html".

[0093] In some embodiments, the crawler framework can be obtained through Scrapy.

[0094] In an embodiment of the present application, in order to simplify the process of writing the crawler framework code and improve the efficiency of crawler writing, the crawler framework code can be pre-stored in a preset configuration database to facilitate users to directly extract it when writing the crawler, thereby simplifying the process of crawler code writing and improving efficiency.

[0095] Among them, in the embodiment of the present application, the method for determining the crawler seed of the crawler task to be crawled according to the configuration information may include:

[0096] Determine task scheduling information based on configuration information;

[0097] According to the task scheduling information, determine the execution priority of the crawler task to be executed and the target crawler node for executing the crawler task to be executed, wherein the target crawler node is a node among the crawler nodes that meets the load requirements, and the crawler node is a node in the crawler crawling system used to execute the crawler task;

[0098] After the target crawler node receives the crawler execution instruction, the execution type of the crawler execution instruction is determined according to the crawler execution instruction, and the execution type includes a new task type and a breakpoint resume task type;

[0099] According to the execution type, the crawler seed of the task to be crawled is obtained.

[0100] The task scheduling information may include the task priority of the crawler task to be configured by the user in the configuration information, the task scheduling cycle configuration, and the information of the designated running node.

[0101] The execution priority may refer to the execution order of the tasks to be crawled. The higher the execution priority, the higher the priority of the tasks to be crawled. The execution priority may be determined according to the task priority in the configuration information. In an embodiment of the present application, the crawler crawling system may read the task pool list in MySQL (relational database management system) in a loop. When the current time meets the scheduling cycle rules, the crawler crawling system obtains the current task to be run. When there are two tasks to be run at the same time, the execution priority of the tasks is obtained and sorted in sequence so that the task with the higher execution priority is executed first.

[0102] A crawler node may refer to a node used to perform crawler tasks. In an embodiment of the present application, there may be multiple crawler nodes in the crawler capture system.

[0103] The target crawler node may refer to a crawler node whose current load meets the preset load requirement among the crawler nodes. The target crawler node is used to execute the tasks to be crawled, wherein the current load may refer to the number of tasks to be executed of the crawler node, and the preset load requirement may refer to the requirement for the number of tasks to be executed. In an embodiment of the present application, the crawler capture system may obtain the number of tasks of each current crawler node, and determine the target crawler node with the least number of tasks based on the number of tasks of each current crawler node, thereby establishing rpc (Remote Procedure Call, Remote Procedure Call Protocol) communication with the target crawler node, and sending the tasks to be crawled to the target crawler node, waiting for scheduling.

[0104] The crawler execution instruction may be an instruction sent by a user to instruct the crawler task to be executed. After receiving the crawler execution instruction, the crawler seed of the crawler task to be executed may be obtained according to the execution type of the crawler execution instruction. Among them, the execution type may include a new task type and a breakpoint resume task type.

[0105] Among them, in the embodiment of the present application, when the execution type is a new task type, according to the execution type, the method for obtaining the crawler seed of the task to be crawled may include:

[0106] Determine, according to the execution type, iteration information in the configuration information and an iteration type of the iteration information;

[0107] According to the iteration type, a link list of tasks to be crawled is obtained, wherein the link list includes an initial crawler seed;

[0108] The initial crawler seed is marked to obtain the crawler seed of the crawler task to be completed.

[0109] The iteration information may include information characterizing link iteration and information characterizing data parameter iteration. When the iteration information exists, the iteration type may be determined based on the iteration information. The iteration type may include a digital iteration type and a file iteration type. When the iteration type is a digital iteration type, the corresponding link list may be generated in batches based on the digital interval. When the iteration type is a file iteration type, the link list may be generated based on the content of each line in the file and the link combination in the crawler configuration information.

[0110] Identification processing may refer to the process of assigning an identity to the initial crawler seed in the link list, wherein the identity may characterize the global uniqueness of the crawler seed. In the embodiment of the present application, the assignment of the identity may be achieved through MD5 (Message-Digest 5), MD4 (Message-Digest 4), SHA-1 (Secure Hash Algorithm 1), etc. The fingerprint may be used in the subsequent data extraction and deduplication module to avoid repeated crawling of the crawler, improve efficiency, and save performance.

[0111] After the identification process is completed, the crawler seed of the crawler task to be crawled is obtained.

[0112] Among them, in the embodiment of the present application, when the execution type is a breakpoint resume crawling task type, according to the execution type, the method for generating a crawler seed for the task to be crawled may include:

[0113] According to the execution type, the crawler seed is read in the crawler task to be crawled, and the crawler seed of the crawler task to be crawled is obtained.

[0114] S102: Execute the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result.

[0115] The current crawler crawling result may refer to the result obtained by executing the crawler task to be executed.

[0116] The method for obtaining the current crawler crawl result may include: obtaining the current crawler request according to the current crawler seed, executing the pending crawler task by sending the current crawler request, and obtaining the current crawler crawl result. In an embodiment of the present application, the seed can be obtained from the seed pool generated by the current task through the load_seeds method, and then the current crawler request request is generated through the make_request method.

[0117] In the embodiment of the present application, before the current crawler request is issued, the proxy configuration, request header information and cookies information can be read and set. After the reading and setting are completed, the current crawler request can be issued to obtain the current crawler crawling results.

[0118] S103: Determine the current crawling depth of the current crawler crawling result according to the configuration information.

[0119] Among them, the current crawling depth can represent that the web crawler will start from the starting page and track one link at a time until the current number of links.

[0120] Among them, in the embodiment of the present application, according to the configuration information, the method for determining the current crawling depth of the current crawler crawling result may include:

[0121] According to the configuration information, determine the current iteration crawling depth of the crawler task to be crawled;

[0122] According to the current iteration crawl depth, determine the current crawl depth of the current crawler crawl results.

[0123] The current iterative crawling depth may be the currently set depth. In the embodiment of the present application, the current iterative crawling depth may be obtained according to the iterative crawling depth in the configuration information. After executing the crawler task once and obtaining the current crawler crawling result, the current iterative crawling depth may be increased by one to obtain the current crawling depth.

[0124] S104: If the current crawling depth does not meet the preset target crawling depth requirement, the next crawler seed and the next crawler request are obtained according to the current crawling result of the crawler, and the next crawler request is a crawler request obtained according to the configuration information and the next crawler seed.

[0125] The target grasping depth requirement may refer to a requirement for the grasping depth. In an embodiment of the present application, the target grasping depth requirement may be preset, for example, the target grasping depth requirement may be 100. When the current grasping depth is 90, the current grasping depth does not meet the preset target grasping depth requirement.

[0126] The next crawler seed may refer to a crawler seed extracted from the current crawler crawl result. The next crawler request may refer to a crawler request obtained according to the configuration information and the next crawler seed.

[0127] Among them, in the embodiment of the present application, if the current crawling depth does not meet the preset target crawling depth requirement, then according to the current crawler crawling result, the method after obtaining the next crawler seed and the next crawler request may also include:

[0128] Get the crawler seed to be executed of the crawler task to be executed in the crawler node;

[0129] Compare the seed of the crawler to be executed with the seed of the next crawler to obtain a comparison result;

[0130] If the comparison result indicates that the crawler seed to be executed and the next crawler seed do not match, then executing the step of obtaining the next crawler request according to the configuration information and the next crawler seed;

[0131] If the comparison result indicates that the crawler seed to be executed matches the next crawler seed, the next crawler seed is deleted, and the crawler seed to be executed is used as the next crawler seed, and the step of obtaining the next crawler request according to the configuration information and the next crawler seed is executed.

[0132] The crawler task to be executed may refer to the crawler task to be executed in the crawler node. The crawler seed to be executed may refer to the crawler seed obtained according to the configuration information of the crawler task to be executed.

[0133] The comparison result may refer to the result of whether the crawler seed to be executed and the next crawler seed are the same. Among them, when the crawler seed to be executed and the next crawler seed are the same, that is, the comparison result indicates that the crawler seed to be executed and the next crawler seed match, then the crawler seed to be executed in the crawler node that matches the next crawler seed can be used as the next crawler seed to execute the step of obtaining the next crawler request according to the configuration information and the next crawler seed, thereby obtaining the next crawler crawling result, thereby improving the efficiency of the crawler node processing. When the crawler seed to be executed and the next crawler seed are not the same, that is, the comparison result indicates that the crawler seed to be executed and the next crawler seed do not match, then the next crawler seed is sent to the crawler node, and the step of obtaining the next crawler request according to the configuration information and the next crawler seed is executed, thereby obtaining the next crawler crawling result.

[0134] S105, using the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, executing the pending crawler task according to the current crawler seed and the current crawler request, and obtaining the steps of the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and obtaining the target crawler crawling result.

[0135] Among them, there can be multiple target crawler crawling results. In an embodiment of the present application, whenever the current crawling depth does not meet the preset target crawling depth requirement, the current crawler crawling result can be stored as the target crawler crawling result until the current crawling depth meets the preset target crawling depth requirement, and multiple target crawler crawling results can be obtained. Among them, in some embodiments, there may be a current crawling depth that does not meet the target crawling depth requirement and is the maximum depth of the crawler task to be crawled. Therefore, when the next crawler seed and the next crawler request are used as the current crawler seed and the current crawler request, the next crawler crawling result obtained is the same as the current crawler crawling result. Therefore, the target crawler crawling result can be deduplicated, thereby obtaining the deduplicated target crawler crawling result.

[0136] Among them, in the embodiment of the present application, after executing the pending crawler task according to the current crawler seed and the current crawler request and obtaining the current crawler crawling result, the method may also include:

[0137] Verify the current crawler crawling results and obtain the verification results;

[0138] If the verification result indicates that the current crawler crawling result is normal, the current crawler crawling result is saved;

[0139] If the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed.

[0140] Among them, the verification result can refer to the judgment of the current crawler crawling result through the abnormal situation set in the check function. If the current crawler crawling result is inconsistent with the abnormal situation, it indicates that the current crawler crawling result is normal, that is, the current crawler crawling result can be saved as the target crawler crawling result, and it can be determined whether to perform the next round of iteration by judging that the current crawling depth does not meet the preset target crawling depth requirement; if the current crawler crawling result is consistent with the abnormal situation, it indicates that the current crawler crawling result is abnormal, and the current crawler seed and the current crawler request can be used to re-execute the pending crawler task and re-obtain the current crawler crawling result.

[0141] Among them, in the embodiment of the present application, if the verification result indicates that the current crawler crawling result is abnormal, the method of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result may include:

[0142] If the verification result indicates that the current crawler crawling result is abnormal, the number of retries for the current crawler crawling result is obtained;

[0143] If the number of retries does not meet the preset retry number threshold requirement, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed;

[0144] If the number of retries meets the preset retry number threshold requirement, it is determined that the status of the current crawler crawling result is abnormal, and the execution of the pending crawler task is terminated.

[0145] The number of retries may refer to the number of times the current crawler crawling result is obtained again after the current crawler crawling result is abnormal.

[0146] The retry count threshold requirement may refer to a preset number of times that the current crawler crawling result can be retrieved after the current crawler crawling result is abnormal.

[0147] After determining that the number of retries does not meet the preset retry threshold requirement, that is, when the number of retries does not reach the preset retry number, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result can be executed.

[0148] After determining that the number of retries meets the preset retry number threshold requirement, that is, when the number of retries reaches the preset retry number, the status of the current crawler crawling result can be marked as abnormal, and the execution of the pending crawler task is terminated.

[0149] The crawler crawling method provided in the present application can determine the current crawling depth of the current crawler result through the configuration information of the crawler task, and quickly determine whether the crawler task is completed based on the comparison between the current crawling depth and the preset target crawling depth requirement, so as to perform crawler crawling according to the next crawler seed and the next crawler request. In this way, it gets rid of the fixed process method in the traditional method, reduces the workload of the staff, makes the crawler logic more flexible, reduces the amount of code, simplifies the process, and achieves the effect of improving efficiency.

[0150] Figure 2 A flowchart of another crawler crawling method provided in an embodiment of the present application, the execution subject of the method can be a computer, and this embodiment is not particularly limited here, such as Figure 2 As shown, the method may include:

[0151] S201. Obtain the name and homepage address of the target website, and create a new data source in the crawler system according to the name and homepage address of the target website.

[0152] The data source is named: XXXXXX;

[0153] The homepage address is: http: / / china.xxxxxxx.com.cn.

[0154] S202: According to the homepage address, the crawler entry and task are configured for the data source to obtain the crawler crawling task.

[0155] The configuration of the crawler entry for the data source can be:

[0156] The corresponding value filled in the entry name can be: XX Daily Current Affairs News;

[0157] The corresponding value to fill in the target crawl URL can be:

[0158] http: / / china.xxxxxxx.com.cn / 5bd5639ca3101a87ca8ff636 / page_1.html;

[0159] If the request type of the target page is GET and the request data is empty, you can set the data to empty.

[0160] The corresponding values ​​of iterative crawling depth can be: 1;

[0161] The corresponding value of the xpath rule for locating the details list can be: / / div[@class="busBox3"];

[0162] The corresponding values ​​of the deeply maintained list positioning xpath rule can be:

[0163] / / *[@id="div_currpage"];

[0164] The corresponding value of the saved page URL regular expression can be:

[0165] china.xxxxxxx.com.cn / a / \d+ / \d+ / .+.html;

[0166] Thus, the current page configuration is completed and saved.

[0167] The configuration of the data source task can be:

[0168] Select the crawler entry under the data source, fill in the task name: xx Daily Current Affairs News, scheduled task, set the task priority to 1, configure the task scheduling cycle to 0****, and specify the running node as random. Among them, the task scheduling cycle can indicate that the current task is run once an hour, and setting the running node to random means assigning any crawler execution node.

[0169] S203: Determine the execution priority of the crawler crawling task and the crawler node that executes the crawler crawling task according to the configuration of the task.

[0170] Among them, the crawler system cyclically reads the task pool list in MySQL. When the current time meets the scheduling cycle rules, the crawler system obtains the current task to be run. If there are two tasks to be run at the same time, they are sorted in sequence according to the scheduling priority of the tasks. In this way, the execution priority of the crawler task can be determined.

[0171] The crawler system can obtain the current task number of each crawler node, and determine the crawler node with lower current load through the task number, thereby establishing RPC communication and sending the crawler crawling task to the crawler node, waiting for scheduling.

[0172] S204: After the crawler node receives the crawler execution command sent by the scheduling node, the crawler node determines the execution type of the crawler capture task according to the type of the crawler execution command.

[0173] Among them, the execution types include: new tasks and breakpoint resume tasks.

[0174] Among them, when it comes to new tasks, the crawler system can read the crawler configuration corresponding to the crawler entrance according to the incoming entrance_id.

[0175] Among them, in some embodiments, for the crawler configuration structure, based on the abstraction of various situations of http requests, in addition to the GET type shown in step S202, it can also support various situations of mainstream get / post requests by judging data and data_type (indicating json type or data type). For example: when the system inputs data that is not empty and data_type is json, it is judged that the current request is a post request of json type; when the system inputs data that is not empty and data_type is raw, it is judged that the current request is a post request of raw type; when the system inputs data that is empty, it is judged that the current request is a regular get type; and the contents of the URL, headers, and cookies fields are judged, and if the content is empty, the default value is filled in, and if it is not empty, the corresponding field content is filled in.

[0176] In the embodiment of the present application, in the process of generating crawler seeds, batch links can be generated based on the URL input by the user and according to the link or data parameter iteration attribute, wherein the process can be:

[0177] Get whether there is link iteration or data parameter iteration (URL_iter or data_iter) in the crawler configuration. If so, determine its iteration type.

[0178] If it is a numeric iteration type, the corresponding link list is generated in batches according to the numeric interval.

[0179] If it is a file iteration type, a link list is generated based on the content of each line in the file and the link combination in the crawler configuration.

[0180] Assign an identity to each crawler seed in the link list, so as to characterize the global uniqueness of the crawler seed. The identity assignment can be achieved through MD5, MD4, SHA-1, etc. The identity assignment method can be used for data extraction and deduplication in subsequent steps, thereby avoiding repeated crawling of the crawler, improving efficiency, and saving performance.

[0181] When the identity is assigned, the seed representing the crawler's crawling task has been generated.

[0182] Among them, for breakpoint resume type tasks: according to the incoming task_id, the crawler system can read all crawler seeds under the task, and then rerun the task. Among them, this function can be used for breakpoint resume of generated seed tasks, or retry of the entire crawler task.

[0183] S205: After obtaining the crawler seed, execute the crawler crawling task to obtain the crawler crawling result.

[0184] Among them, you can use the load_seeds method to obtain crawler seeds from the seed pool generated by the crawler crawling task, and use the make_request method to generate a request.

[0185] In an embodiment of the present application, the proxy configuration, request header information and cookies information can be read and set before the request is sent. After the request is sent, the response content can be obtained to obtain the crawler crawling results.

[0186] S206: Verify the crawler crawling results, and determine whether to save the crawler crawling results based on the verified results.

[0187] Among them, you can use the code:

[0188] def validate(self, request, response):

[0189] flag = check(response);

[0190] return flag.

[0191] Verify the crawler crawling results, where the return value of flag can be True or False. The check function can be set to define the abnormal situation, so as to determine whether the result after verification is abnormal. That is, if it is consistent with the built-in default abnormal situation, it indicates an abnormality, indicating that the result after verification is abnormal, and the Flag return value is False. If there is a custom forbidden_words or forbidden_feature_xpath value as the standard value for result verification, it is compared in sequence, indicating that the result after verification is normal. If it is consistent with the standard value, it indicates an abnormality, indicating that the result after verification is abnormal, and the Flag return value is False.

[0192] In some embodiments, when the Flag return value is False, it is determined whether the current seed has reached the maximum number of retries. If not, it is re-executed; otherwise, the current seed execution action is terminated and the seed status is marked as abnormal.

[0193] After determining that the crawler crawling results are normal and need to be saved, you can call the page saver to perform the save operation.

[0194] S207: Determine the current crawling depth value of the crawler crawling result, and determine whether the crawler crawling task is completed according to the current crawling depth value.

[0195] The current crawling depth value may be determined according to the corresponding value of the iterative crawling depth in the configuration of the crawler entry in step S202 , for example, the current crawling depth value=the corresponding value of the iterative crawling depth+1.

[0196] Among them, after obtaining the current crawling depth value, it can be determined whether the current crawling depth reaches the maximum crawling depth. If not, link extraction is performed according to the extraction rules to obtain a new crawler seed, and the crawler crawling result is marked as normal, and the execution action is terminated. For the extracted new crawler seeds, the seed status is marked as unscheduled, and it is placed in the scheduling pool waiting for scheduling. At the same time, the identity of the extracted new crawler seeds is compared in the scheduling pool. If it already exists, the crawler seed is discarded.

[0197] Figure 3 A schematic diagram of the module flow in the crawler system provided in the embodiment of the present application, such as Figure 3 As shown, the crawler system may include:

[0198] Backend management module, information configuration module, task scheduling module, crawler module, request information management module, data extraction and deduplication module, result verification module and page saving module.

[0199] Among them, the background management module is used to retry abnormal crawlers, view and export data of acquired pages, manage the status of crawlers, and manage the status of tasks.

[0200] The information configuration module is used to configure the data source information, wherein the configured data source information includes the data source name, homepage address, etc., and configuration entry information; fill in the configuration information based on the target page, wherein the filled configuration information includes: setting the URL, headers, cookies of the target page, the list positioning of the page, the positioning of the page turning elements, and other information.

[0201] The task scheduling module is used to schedule the crawler tasks based on the scheduling cycle parameters of the task configuration. The functions of the task scheduling module may include: setting the scheduling cycle, scheduling interval, and scheduling priority of the target task, obtaining the current idle rate of each crawler node, and whether the data crawled by the current task is duplicated in the database.

[0202] The crawler module is used to obtain the source code of the target page. The content obtained may include: rescheduling the request seeds whose number of attempts in the current page is less than the maximum number of retries, marking the seed status after the request, and obtaining the webpage source code returned by the website.

[0203] The request information management module is used to obtain the input information of the current task and output it to the crawler module. The output content may include initializing request parameters to generate seeds, resetting seed status, inputting and filling in the proxy, cookies, and headers information during the crawler process, deep marking of crawler requests, and information transmission between requests.

[0204] The data extraction and deduplication module is used to crawl pages that need incremental crawling according to rules based on preset task configurations, filter and deduplicate repetitive requests, and improve system performance.

[0205] The result verification module is used to verify the obtained target page based on the configuration rules in the information configuration module, mark the seed status of invalid request pages and return them to the crawler module for retry.

[0206] The page saving module is used to persistently store web pages that have passed page verification and save the source code of successfully captured web pages, response status code, response URL, storage time, and data source information corresponding to the data. Figure 4 A flow chart of the scheduler operation logic provided in the embodiment of the present application, such as Figure 4 As shown, the process may include:

[0207] Load system components; start the scheduled scheduling service, obtain the crawler configuration, crawler task configuration, and connect the crawler sub-nodes; thus, the scheduler service is started, and then all scheduled tasks are loaded, so that the scheduler starts running, and determines whether there are tasks to be run in the scheduler. If not, return to the process where the scheduler starts running. If so, obtain the task configuration. If the task configuration is not obtained, it can return to the process where the scheduler starts running. After obtaining the task configuration, perform node communication and send tasks.

[0208] Figure 5 A flow chart of the grabber logic provided in the embodiment of the present application is as follows: Figure 5 As shown, the process may include:

[0209] Receive the scheduler command and determine the task execution type; if the incoming task execution type is entrance_id, obtain the corresponding configuration of the crawler task, generate a request task, determine the request link type, and when the request link type is a single link, generate a request seed, and set different types of requests (get / post) according to the incoming parameters, and then set proxies, headers, and cookies. Then the requester schedules the request, obtains the response, adds depth information, and obtains the result. After obtaining the result, the result verifier determines whether the content is normal. If normal, determine whether it needs to be saved. If not, determine whether it needs to be extracted. If not, mark the seed status as completed.

[0210] Among them, if the incoming task execution type is task_id, the process of generating request seeds is executed; when the request link type is multiple links, multiple request seeds are generated according to the configuration, and the process of setting different types of requests (get / post) according to the incoming parameters is executed; when the verifier determines that the content is abnormal, the process of scheduling the request is returned to the requester; if it is determined that it needs to be saved, the page save operation is performed, and the process of determining whether extraction is required is executed; if extraction is required, the target link is extracted, and the process of generating request seeds is returned.

[0211] Another crawler crawling method provided by the embodiment of the present application can modularize the components such as the task management, request management, data source management, and database connection of the crawler, and integrate them into the system, so that users no longer need to pay too much attention to issues such as database connection, file management of crawlers of various data sources, and scheduling management of various crawler tasks, etc., for the situation that the existing framework design concept is relatively complex and difficult to get started. By pre-implanting default settings in the system, the user is shielded from the conventional code writing process such as the use of proxies in the crawler process, automatic resending of network request exceptions, and automatic retry mechanism for returning common abnormal pages. It only requires users to have a basic understanding of web page parsing and know xpath syntax or regular syntax. It allows users to not pay attention to some repetitive and tedious work content; finally, for the situation that the threshold for the conventional crawler code writing process is high and there are many repetitive tasks after mastering it, users only need to complete the configuration writing to complete the configuration of the crawler process, without having to write a line of code, thereby reducing the workload, thereby solving the problem that the existing crawler crawling method has a relatively cumbersome process and low efficiency.

[0212] Figure 6 This is a schematic diagram of the structure of the crawler grabbing device provided in the embodiment of the present application. Figure 5 As shown, the crawler grasping device 60 includes: an acquisition module 601, an execution module 602, a determination module 603, a first obtaining module 604 and a second obtaining module 605. Among them:

[0213] The acquisition module 601 is used to obtain the configuration information of the crawler task to be performed, the current crawler seed and the current crawler request;

[0214] An execution module 602 is used to execute the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result;

[0215] A determination module 603 is used to determine the current crawling depth of the current crawler crawling result according to the configuration information;

[0216] The first obtaining module 604 is used to obtain the next crawler seed and the next crawler request according to the current crawling result if the current crawling depth does not meet the preset target crawling depth requirement, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed;

[0217] The second obtaining module 605 is used to use the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, execute the pending crawler task according to the current crawler seed and the current crawler request, and obtain the steps of the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and obtains the target crawler crawling result.

[0218] In the embodiment of the present application, the acquisition module 601 may also be specifically used for:

[0219] In response to the user's configuration operation, obtain the configuration information of the crawler task to be processed;

[0220] Determine the crawler seed for the crawler task to be completed based on the configuration information;

[0221] Get the crawler request based on the crawler seed and configuration information.

[0222] In the embodiment of the present application, the acquisition module 601 may also be specifically used for:

[0223] In response to the user's template generation operation, a configuration template corresponding to the crawler task to be performed is obtained, where the configuration template includes a plurality of configuration units, and the configuration units are units selected by the user from a preset configuration database according to the crawler task to be performed;

[0224] In response to the user's write operation on the configuration unit according to the task to be crawled, the configuration information of the task to be crawled is obtained.

[0225] In the embodiment of the present application, the acquisition module 601 may also be specifically used for:

[0226] Determine task scheduling information based on configuration information;

[0227] According to the task scheduling information, determine the execution priority of the crawler task to be executed and the target crawler node for executing the crawler task to be executed, wherein the target crawler node is a node among the crawler nodes that meets the load requirements, and the crawler node is a node in the crawler crawling system used to execute the crawler task;

[0228] After the target crawler node receives the crawler execution instruction, the execution type of the crawler execution instruction is determined according to the crawler execution instruction, and the execution type includes a new task type and a breakpoint resume task type;

[0229] According to the execution type, the crawler seed of the task to be crawled is obtained.

[0230] In the embodiment of the present application, the acquisition module 601 may also be specifically used for:

[0231] Determine, according to the execution type, iteration information in the configuration information and an iteration type of the iteration information;

[0232] According to the iteration type, a link list of tasks to be crawled is obtained, wherein the link list includes an initial crawler seed;

[0233] The initial crawler seed is marked to obtain the crawler seed of the crawler task to be completed.

[0234] In the embodiment of the present application, the acquisition module 601 may also be specifically used for:

[0235] According to the execution type, the crawler seed is read in the crawler task to be crawled, and the crawler seed of the crawler task to be crawled is obtained.

[0236] In the embodiment of the present application, the determination module 603 may also be specifically used for:

[0237] According to the configuration information, determine the current iteration crawling depth of the crawler task to be crawled;

[0238] According to the current iteration crawl depth, determine the current crawl depth of the current crawler crawl results.

[0239] In the embodiment of the present application, the determination module 603 may also be specifically used for:

[0240] If the current crawling depth meets the preset target crawling depth requirement, all crawling results of the crawler executing the crawler task are obtained;

[0241] Determine the target crawler crawling results based on all crawler crawling results.

[0242] In the embodiment of the present application, the first obtaining module 604 may also be specifically used for:

[0243] Get the crawler seed to be executed of the crawler task to be executed in the crawler node;

[0244] Compare the seed of the crawler to be executed with the seed of the next crawler to obtain a comparison result;

[0245] If the comparison result indicates that the crawler seed to be executed and the next crawler seed do not match, then executing the step of obtaining the next crawler request according to the configuration information and the next crawler seed;

[0246] If the comparison result indicates that the crawler seed to be executed matches the next crawler seed, the next crawler seed is deleted, and the crawler seed to be executed is used as the next crawler seed, and the step of obtaining the next crawler request according to the configuration information and the next crawler seed is executed.

[0247] In the embodiment of the present application, the execution module 602 may also be specifically used for:

[0248] Verify the current crawler crawling results and obtain the verification results;

[0249] If the verification result indicates that the current crawler crawling result is normal, the current crawler crawling result is saved;

[0250] If the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed.

[0251] In the embodiment of the present application, the execution module 602 may also be specifically used for:

[0252] If the verification result indicates that the current crawler crawling result is abnormal, the number of retries for the current crawler crawling result is obtained;

[0253] If the number of retries does not meet the preset retry number threshold requirement, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is executed;

[0254] If the number of retries meets the preset retry number threshold requirement, it is determined that the status of the current crawler crawling result is abnormal, and the execution of the pending crawler task is terminated.

[0255] As can be seen from the above, the crawler crawling device of this embodiment is composed of an acquisition module 601, which is used to obtain the configuration information of the task to be crawled, the current crawler seed and the current crawler request; an execution module 602, which is used to execute the task to be crawled according to the current crawler seed and the current crawler request, and obtain the current crawler crawling result; a determination module 603, which is used to determine the current crawling depth of the current crawler crawling result according to the configuration information; a first acquisition module 604, which is used to obtain the next crawler seed and the next crawler request according to the current crawler crawling result if the current crawling depth does not meet the preset target crawling depth requirement, and the next crawler request is the crawler request obtained according to the configuration information and the next crawler seed; a second acquisition module 605, which is used to use the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, and execute the steps of executing the task to be crawled according to the current crawler seed and the current crawler request to obtain the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and the target crawler crawling result is obtained; thereby, the effect of improving efficiency is achieved.

[0256] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 70 includes:

[0257] The electronic device 70 may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a communication component 703 and other components. The processor 701 , the memory 702 and the communication component 703 are connected via a bus 704 .

[0258] In the specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the above crawler crawling method.

[0259] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0260] In the above Figure 7 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0261] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0262] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0263] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the steps in any of the above-mentioned crawler crawling methods when executed by a processor.

[0264] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0265] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0266] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any crawler crawling method provided in the embodiment of the present application.

[0267] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0268] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0269] Since the instructions stored in the storage medium can execute the steps in any crawler crawling method provided in the embodiments of the present application, the beneficial effects that can be achieved by any crawler crawling method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0270] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0271] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A crawler crawling method, characterized in that: Applied to a crawler grasping system, the method comprises: Get the configuration information of the crawler task to be crawled, the current crawler seed and the current crawler request; According to the current crawler seed and the current crawler request, the pending crawler task is executed to obtain the current crawler crawling result; Determine the current crawling depth of the current crawler crawling result according to the configuration information; If the current crawling depth does not meet the preset target crawling depth requirement, then according to the current crawler crawling result, a next crawler seed and a next crawler request are obtained, and the next crawler request is a crawler request obtained according to the configuration information and the next crawler seed; The next crawler seed and the next crawler request are used as the current crawler seed and the current crawler request, and the steps of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result are performed until the current crawling depth meets the preset target crawling depth requirement, and the target crawler crawling result is obtained.

2. The method according to claim 1, characterized in that The step of obtaining the configuration information of the crawler task to be processed, the current crawler seed, and the current crawler request includes: In response to the user's configuration operation, obtain the configuration information of the crawler task to be processed; Determine the crawler seed of the crawler task to be crawled according to the configuration information; A crawler request is obtained according to the crawler seed and the configuration information.

3. The method according to claim 2, characterized in that The step of obtaining configuration information of the crawler task to be performed in response to the configuration operation of the user includes: In response to a template generation operation by a user, a configuration template corresponding to the crawler task to be performed is obtained, wherein the configuration template includes a plurality of configuration units, and the configuration units are units selected by the user from a preset configuration database according to the crawler task to be performed; In response to a user's write operation on the configuration unit according to the task to be crawled, configuration information of the task to be crawled is obtained.

4. The method according to claim 2, characterized in that: Determining the crawler seed of the crawler task to be crawled according to the configuration information includes: Determine task scheduling information according to the configuration information; Determine the execution priority of the task to be crawled and the target crawler node for executing the task to be crawled according to the task scheduling information, wherein the target crawler node is a node among the crawler nodes that meets the load requirement, and the crawler node is a node in the crawler crawling system for executing the crawler task; After the target crawler node receives the crawler execution instruction, the execution type of the crawler execution instruction is determined according to the crawler execution instruction, and the execution type includes a new task type and a breakpoint resume task type; According to the execution type, a crawler seed of the task to be crawled is obtained.

5. The method according to claim 4, characterized in that When the execution type is a new task type, obtaining the crawler seed of the task to be crawled according to the execution type includes: Determine, according to the execution type, iteration information in the configuration information and an iteration type of the iteration information; According to the iteration type, a link list of the pending crawler tasks is obtained, wherein the link list includes an initial crawler seed; The initial crawler seed is marked to obtain the crawler seed of the crawler task to be performed.

6. The method according to claim 4, characterized in that When the execution type is a breakpoint resume task type, generating a crawler seed of the task to be crawled according to the execution type includes: According to the execution type, a crawler seed is read in the task to be crawled to obtain a crawler seed of the task to be crawled.

7. The method according to claim 1, characterized in that Determining the current crawling depth of the current crawler crawling result according to the configuration information includes: Determine the current iteration crawling depth of the crawler task according to the configuration information; According to the current iterative crawling depth, a current crawling depth of the current crawler crawling result is determined.

8. The method according to claim 1, characterized in that After determining the current crawling depth of the current crawler crawling result according to the configuration information, the method further includes: If the current crawling depth meets the preset target crawling depth requirement, all crawling results of executing the crawler task to be crawled are obtained; According to all the crawler crawling results, the target crawler crawling results are determined.

9. The method according to claim 1, characterized in that: If the current crawling depth does not meet the preset target crawling depth requirement, then after obtaining the next crawler seed and the next crawler request according to the current crawler crawling result, the method further includes: Get the crawler seed to be executed of the crawler task to be executed in the crawler node; Comparing the to-be-executed crawler seed with the next crawler seed to obtain a comparison result; If the comparison result indicates that the to-be-executed crawler seed and the next crawler seed do not match, executing the step of obtaining a next crawler request according to the configuration information and the next crawler seed; If the comparison result indicates that the crawler seed to be executed matches the next crawler seed, the next crawler seed is deleted, and the crawler seed to be executed is used as the next crawler seed to execute the step of obtaining the next crawler request according to the configuration information and the next crawler seed.

10. The method according to claim 1, characterized in that After executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result, the method further includes: Verify the current crawler crawling result to obtain a verification result; If the verification result indicates that the current crawler crawling result is normal, then the current crawler crawling result is saved; If the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result is performed.

11. The method according to claim 10, characterized in that If the verification result indicates that the current crawler crawling result is abnormal, the step of executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result includes: If the verification result indicates that the current crawler crawling result is abnormal, the number of retries of the current crawler crawling result is obtained; If the number of retries does not meet the preset retry number threshold requirement, executing the pending crawler task according to the current crawler seed and the current crawler request to obtain the current crawler crawling result; If the number of retries meets the preset retry number threshold requirement, it is determined that the state of the current crawler crawling result is abnormal, and the execution of the pending crawler task is terminated.

12. A crawler grabbing device, characterized in that: include: The acquisition module is used to obtain the configuration information of the crawler task to be processed, the current crawler seed, and the current crawler request; An execution module, used for executing the pending crawler task according to the current crawler seed and the current crawler request, and obtaining the current crawler crawling result; A determination module, used to determine the current crawling depth of the current crawler crawling result according to the configuration information; A first obtaining module is used to obtain a next crawler seed and a next crawler request according to the current crawler crawling result if the current crawling depth does not meet the preset target crawling depth requirement, wherein the next crawler request is a crawler request obtained according to the configuration information and the next crawler seed; The second obtaining module is used to use the next crawler seed and the next crawler request as the current crawler seed and the current crawler request, execute the steps of executing the pending crawler task according to the current crawler seed and the current crawler request, and obtain the current crawler crawling result, until the current crawling depth meets the preset target crawling depth requirement, and obtains the target crawler crawling result.

13. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the crawler crawling method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the crawler crawling method according to any one of claims 1 to 11.