Crawler task error processing method and device

Abstract

The present invention discloses a method and device for processing crawler task errors, which relates to the field of crawler technology, and is mainly used to solve the problem of excessively high labor costs in the current processing of crawler task errors. The method comprises: after obtaining the error reporting task, grouping the error reporting task according to the task parameters to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task; executing the error reporting task according to the task group by the crawler, and obtaining the grouping task result of each task group; judging whether the error reporting task is executed successfully according to the grouping task result; if unsuccessful, determining the error reporting task as an abnormal task and outputting it. The present invention is used for the processing of crawler task errors.

Description

Technical Field

[0001] The present invention relates to the field of crawler technology, and in particular to a method and device for processing crawler task errors. Background Art

[0002] With the explosive growth of Internet data, the way of obtaining data has also changed. Among them, crawlers, as a means of computer automated crawling of web content information, have gradually become popular. With the development of front-end technology and the popularization of distributed services, the loading methods of web pages and the presentation of content have become more diverse and complex. This makes it very easy for crawling to fail due to web page opening errors or network problems when using crawlers to crawl web content.

[0003] At present, after the crawler executes the crawling task, it will feedback error information when the crawling fails. Generally speaking, the handling method of crawler task errors is often based on manual methods. That is to say, when a crawler task error is detected, relevant staff are required to manually determine whether the task that reports the error is indeed a task that the crawler cannot crawl, or whether it is simply because the webpage failed to open correctly during the crawling process, or there are interferences from network factors during the crawling. However, in actual applications, the existing handling method of crawler task errors is too dependent on manual operations, especially for some errors that can be solved by reopening the website and refreshing it. The existing handling method will bring about the problem of excessively high labor costs through manual operations. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and device for processing crawler task errors, the main purpose of which is to solve the problem of high labor costs in the current processing of crawler task errors.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a method for processing crawler task errors, the method comprising:

[0006] After obtaining the error reporting tasks, the error reporting tasks are grouped according to the task parameters to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task;

[0007] Execute the error reporting task according to the task grouping by a crawler, and obtain the grouping task result of each task group;

[0008] According to the grouping task result, judging whether the error reporting task is executed successfully;

[0009] If unsuccessful, the error reporting task is determined as an abnormal task and outputted.

[0010] Optionally, after obtaining the error reporting tasks, the error reporting tasks are grouped according to task parameters, and the obtained task groups include:

[0011] Acquire task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawling page type information, crawling depth information and error information;

[0012] According to the task parameters, a plurality of the error reporting tasks are allocated into a plurality of the task groups, wherein each of the task groups includes at least one error reporting task having the same task parameters.

[0013] Optionally, before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes:

[0014] Determine whether the number of tasks in each of the task groups reaches a task threshold;

[0015] If not reached, a copy operation is performed on the error-reporting tasks in the task group so that the number of tasks reaches the task threshold.

[0016] Optionally, executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group includes:

[0017] Executing each of the error reporting tasks according to the task groups by the crawler, and obtaining the execution result corresponding to each of the error reporting tasks;

[0018] The execution results of all the error reporting tasks in the task group are counted to obtain the group task result.

[0019] Optionally, judging whether the error reporting task is successfully executed according to the grouping task result includes:

[0020] Determine whether at least one of the plurality of execution results corresponding to the grouped task result is successfully executed;

[0021] If so, determining that the error reporting task corresponding to the grouping task is executed successfully;

[0022] If it does not exist, determining that the error reporting task corresponding to the grouping task fails to execute;

[0023] If the task fails, the error reporting task is determined as an abnormal task and outputted, including:

[0024] If the error reporting task corresponding to the grouping task fails to execute, the error reporting task is determined as an abnormal task and output.

[0025] Optionally, before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes:

[0026] Grouping a plurality of the tasks into a task batch;

[0027] Before determining the error reporting task as an abnormal task and outputting it, the method further includes:

[0028] Determine the task completion rate of each task batch according to the grouped task results, wherein the task completion rate is used to characterize the proportion of successful crawling in the grouped task results in the task batch;

[0029] The step of determining the error reporting task as an abnormal task and outputting the task includes:

[0030] If the task completion rate is lower than a preset ratio threshold, the error-reporting task is determined as an abnormal task and output;

[0031] If the task completion rate is higher than a preset ratio threshold, the error reporting task is added to the task group in the next task batch.

[0032] In a second aspect, an embodiment of the present invention further provides a crawler task error processing device, comprising:

[0033] A grouping unit, for grouping the error reporting tasks according to task parameters after acquiring the error reporting tasks, to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task for which the crawler reports a crawling failure after executing the crawling task;

[0034] An execution unit, used for executing the error reporting task according to the task group through a crawler, and obtaining a group task result of each task group;

[0035] A first judging unit, configured to judge whether the error reporting task is successfully executed according to the grouping task result;

[0036] The first determining unit is used to determine the error reporting task as an abnormal task and output it if it is determined that the error reporting task is not executed successfully.

[0037] Optionally, the grouping unit includes:

[0038] An acquisition module, used for acquiring task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawling page type information, crawling depth information and error information;

[0039] An allocating module is used to allocate the plurality of error reporting tasks into a plurality of task groups according to the task parameters, wherein each of the task groups contains at least one error reporting task with the same task parameters.

[0040] Optionally, the device further comprises:

[0041] A second judging unit, used to judge whether the number of tasks in each of the task groups reaches a task threshold;

[0042] The copying unit is used to perform a copy operation on the error-reporting tasks in the task group if it is determined that the number of tasks in each of the task groups does not reach the task threshold, so that the number of tasks reaches the task threshold.

[0043] Optionally, the execution unit includes:

[0044] An execution module, used to execute each of the error reporting tasks according to the task grouping through the crawler, and obtain an execution result corresponding to each of the error reporting tasks;

[0045] The statistics module is used to count the execution results of all the error reporting tasks in the task group to obtain the group task results.

[0046] Optionally, the first judging unit includes:

[0047] A judgment module, used for judging whether there is at least one execution result that is successfully executed among the multiple execution results corresponding to the grouped task results;

[0048] A first determination module, configured to determine that the error reporting task corresponding to the grouped task is executed successfully if at least one of the plurality of execution results corresponding to the grouped task result is determined to be successfully executed;

[0049] A second determination module is used to determine that the error reporting task corresponding to the grouped task has failed to execute if it is determined that there is not at least one execution result that is successfully executed among the multiple execution results corresponding to the grouped task result;

[0050] The first determining unit is specifically configured to determine and output the error reporting task as an abnormal task if the error reporting task corresponding to the grouping task fails to execute.

[0051] Optionally, the device further comprises:

[0052] A setting unit, used for setting a plurality of the task groups into a task batch;

[0053] The device also includes:

[0054] A second determining unit is used to determine the task completion rate of each task batch according to the grouped task results, wherein the task completion rate is used to represent the proportion of successful crawling in the grouped task results in the task batch;

[0055] The first determining unit includes:

[0056] A first determination module, configured to determine the error-reporting task as an abnormal task and output it if the task completion rate is lower than a preset ratio threshold;

[0057] An adding module is used to add the error reporting task to the task group in the next task batch if the task completion rate is higher than a preset ratio threshold.

[0058] In order to achieve the above-mentioned purpose, according to the third aspect of the present invention, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned crawler task error processing method.

[0059] In order to achieve the above-mentioned purpose, according to a fourth aspect of the present invention, a processor is provided, and the processor is used to run a program, wherein the program executes the above-mentioned method for processing crawler task errors when running.

[0060] Through the above technical scheme, the method and device for processing crawler task errors provided by the present invention can solve the problem of high labor costs in the existing crawler task error processing process. After obtaining the error task, the present invention groups the error task according to task parameters to obtain task groups, and then executes the error task according to the task group by the crawler, and obtains the grouped task results of each task group. Then, according to the grouped task results, it is judged whether the error task is executed successfully. Finally, if it is unsuccessful, the error task is determined as an abnormal task and output, thereby completing the processing of the crawler task error. In the above scheme, each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task. This makes it possible to automatically obtain the task that failed to crawl before when processing the crawler task error, and execute the error reporting task again in the form of grouping based on the task grouping. This ensures that the failed error reporting task can be automatically obtained and executed again after the crawler crawls and fails, and can avoid the error reporting task caused by network or page opening failure. It can be executed again, and there is no need to manually judge whether the crawling failure is caused by the above problems, saving labor costs. At the same time, after executing according to the task grouping, when it is determined that the error reporting task still cannot be crawled successfully, the error reporting task is determined as an abnormal task and output, which can ensure that the user can directly determine that the task is indeed a task that cannot be crawled based on the output result, without the need for manual identification, further saving manpower, thereby solving the problem of excessive labor costs.

[0061] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0063] Figure 1 A flowchart of a method for processing crawler task errors provided by an embodiment of the present invention is shown;

[0064] Figure 2 A flow chart of another method for processing crawler task errors provided by an embodiment of the present invention is shown;

[0065] Figure 3A block diagram showing a composition of a crawler task error processing device provided by an embodiment of the present invention;

[0066] Figure 4 A block diagram showing another apparatus for processing crawler task errors provided by an embodiment of the present invention;

[0067] Figure 5 A block diagram showing the composition of a device for processing crawler task errors provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0069] In order to solve the problem of high labor cost in the current process of processing crawler task errors, an embodiment of the present invention provides a method for processing crawler task errors, such as Figure 1 As shown, the method includes:

[0070] 101. After the error reporting tasks are obtained, the error reporting tasks are grouped according to the task parameters to obtain task groups.

[0071] Each of the task groups includes at least one error reporting task, and the error reporting task is a task in which the crawler reports a crawling failure after executing the crawling task.

[0072] In this embodiment, the error reporting task can be understood as a task in which the crawler reports a crawling failure after executing the crawling task. Since some of these tasks may fail to crawl simply because the web page cannot be opened correctly, these error reporting tasks need to be confirmed again.

[0073] At the same time, since each crawling task has corresponding parameters during crawling, the error reporting task also corresponds to task parameters. In this embodiment, the error reporting tasks can be grouped according to the task parameters to obtain grouped tasks, namely task groups.

[0074] Specifically, the grouping method can be to group the same or similar task error reporting tasks into one group by adjusting the task parameters. The task parameter can be a URL domain name. URL, i.e., uniform resource locator, is a representation method for specifying the location of information on the World Wide Web service program of the Internet. It was originally invented by Tim Berners-Lee as the address of the World Wide Web. Of course, in actual applications, the task parameter can also be other information used to characterize the error reporting task, which is not limited here.

[0075] 102. Execute the error reporting task according to the task grouping through the crawler, and obtain the grouping task result of each task group.

[0076] Since some tasks are only due to network problems when the crawler is crawling, or the page fails to open, such tasks can actually be solved by executing them again. Therefore, in this embodiment, after the error reporting tasks have been grouped according to the task parameters in the previous steps, the above error reporting tasks can be executed again according to the groups using the crawler based on the method of this step. Since the error reporting tasks are executed based on task groups, and the task groups are divided according to the task parameters, the grouping task results of the task groups obtained in this step are actually determined by the crawling results of each error reporting task in the group.

[0077] 103. Based on the grouping task results, determine whether the error reporting task is executed successfully.

[0078] Since in the process of the crawler reporting an error and trying to crawl again, the tasks it executes may be multiple identical tasks with the same task parameters, and therefore they have been divided into a task group in the aforementioned step 101. Therefore, when determining whether the error-reporting task is successfully executed in this step, whether the error-reporting task is successfully executed can be determined based on the grouped task results from the crawling results of multiple tasks.

[0079] For example, when the task group includes error reporting task 1, error reporting task 2, and error reporting task 3. And these three tasks are actually crawling tasks for a target page, then the group task results are error reporting task 1 execution success, error reporting task 2 execution failure, and error reporting task 3 execution failure. At this time, for error reporting tasks 1, 2, and 3, since the three tasks are actually the same, only one of them is successful to determine that the crawler has actually successfully crawled the target page, so that the group of tasks can be considered to have been successfully executed.

[0080] 104. If unsuccessful, the error reporting task is determined as an abnormal task and outputted.

[0081] Based on the judgment of 103, when it is determined that the error reporting task has not been successfully executed, it means that the result of the grouping task shows that the error reporting task crawling of this task group has failed, and because the task grouping is divided according to the task parameters, the error reporting tasks contained in each task group actually correspond to the same crawling target. This means that when all tasks in a task group fail to execute, it can be inferred that this crawling target cannot be completed by the crawler. Based on this, the error reporting task can be determined as an abnormal task and output to the user, so that the user can directly determine that this crawling task cannot be completed by the crawler based on the output abnormal task, and then there is no need to manually identify whether there are other reasons.

[0082] The above-mentioned embodiment provides a method for processing crawler task errors. For the problem of excessively high labor costs in the existing crawler task error processing process, the present invention groups the error tasks according to task parameters after obtaining the error tasks to obtain task groups, and then executes the error tasks according to the task groups by the crawler, and obtains the grouped task results of each task group, and then determines whether the error task is successfully executed based on the grouped task results. Finally, if unsuccessful, the error task is determined as an abnormal task and outputted, thereby completing the processing of crawler task errors. In the above scheme, each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task. This makes it possible to automatically obtain the task that failed to crawl before when processing the crawler task error, and execute the error reporting task again in the form of grouping based on the task grouping. This ensures that the failed error reporting task can be automatically obtained and executed again after the crawler crawls and fails, and can avoid the error reporting task caused by network or page opening failure. It can be executed again, and there is no need to manually judge whether the crawling failure is caused by the above problems, saving labor costs. At the same time, after executing according to the task grouping, when it is determined that the error reporting task still cannot be crawled successfully, the error reporting task is determined as an abnormal task and output, which can ensure that the user can directly determine that the task is indeed a task that cannot be crawled based on the output result, without the need for manual identification, further saving manpower, thereby solving the problem of excessive labor costs.

[0083] Furthermore, as a Figure 1 The embodiment shown in the figure is refined and expanded. The embodiment of the present invention also provides another method for processing crawler task errors, such as Figure 2 As shown, the specific steps include:

[0084] 201. After obtaining the error reporting tasks, group the error reporting tasks according to the task parameters to obtain task groups.

[0085] Each of the task groups includes at least one error reporting task, and the error reporting task is a task in which the crawler reports a crawling failure after executing the crawling task.

[0086] Specifically, this step includes:

[0087] First, the task parameters of the error reporting task are obtained. The task parameters include at least one of uniform resource locator information, crawled page type information, crawling depth information and error information. In this embodiment, the resource locator information may include URL domain name and URL depth, wherein the URL depth can be used to characterize the length and complexity of the URL. The crawled page type information can be used to characterize whether the crawled page belongs to the search page list or a normal page. The crawling depth information is used to characterize the level of the page link crawled by the crawler, and the error information is used to characterize the error identification information fed back during crawling, such as 404, 503, etc.

[0088] Then, according to the task parameters, the plurality of error reporting tasks are allocated to a plurality of task groups. Each of the task groups contains at least one error reporting task with the same task parameters. In this embodiment, the task groups can be divided based on the situation that the above-mentioned task parameters are the same. Of course, the specific operation process is related to the specific task parameters involved. For example, when the IDE task parameters involved only include uniform resource locator information, it is only necessary to ensure that the uniform resource locator information is the same and then allocate it to one task group.

[0089] In practical applications, the above error reporting tasks can be placed in the cache pool after being acquired. Since the data in the cache pool has an efficient reading effect, the efficiency of subsequent operations can be ensured. At the same time, the cache pool can also be used as a judgment process for subsequent detection of whether there are error reporting tasks to be executed.

[0090] 202. Determine whether the number of tasks in each task group reaches a task threshold.

[0091] In order to ensure that during the crawling process, it can be directly determined whether the task that failed to crawl before is indeed the task that the crawler cannot crawl, in this embodiment, the assigned task grouping can also be judged to avoid too few tasks in the task grouping.

[0092] 203. If not reached, a copy operation is performed on the error-reporting tasks in the task group so that the number of tasks reaches the task threshold.

[0093] When the number of tasks determined in the above step does not reach the task threshold, it means that the number of error-reporting tasks in the task group is too small. Therefore, in this step, the error-reporting tasks in this task group can be copied so that the number of tasks in this task group reaches the task threshold. In this way, the number of tasks in each task group can meet expectations, and the copied tasks are exactly the same as the copied tasks, that is, in subsequent executions, it is equivalent to performing one more crawl, so that it can be determined that the task execution results can be more stable and accurate in subsequent executions.

[0094] 204. Group multiple tasks into a task batch.

[0095] In practical applications, the method described in this embodiment can be repeated multiple times, that is, the crawler can be used to perform multiple cycles of crawling on the tasks that failed to crawl. Therefore, after the error reporting tasks are divided into task groups in the aforementioned steps, these task groups can be set up as task batches to ensure that they are executed in batches in subsequent steps.

[0096] 205. The error reporting task is executed by the crawler according to the task group, and the group task result of each task group is obtained.

[0097] Specifically, this step may include:

[0098] First, the crawler executes each of the error reporting tasks according to the task grouping, and obtains the execution result corresponding to each of the error reporting tasks;

[0099] Then, the execution results of all the error reporting tasks in the task group are counted to obtain the group task result.

[0100] In the above execution process, since the task group contains at least one error reporting task, the execution result of each error reporting task can be obtained after execution. That is to say, each task group can obtain multiple execution results after using the crawler to perform the crawling operation. At the same time, since the error reporting tasks in the task group are divided according to task parameters, these tasks are actually tasks of the same crawling target, so the group task results can be determined based on these execution results.

[0101] 206. According to the grouping task result, determine whether the error reporting task is executed successfully.

[0102] In the specific operation process, since the grouping task result may include multiple execution results, in this case, specifically, in actual application, this step may include:

[0103] Determine whether at least one of the plurality of execution results corresponding to the grouped task result is successfully executed;

[0104] If so, determining that the error reporting task corresponding to the grouping task is executed successfully;

[0105] If not, it is determined that the error reporting task corresponding to the grouping task fails to execute.

[0106] In this way, since a task group actually corresponds to a crawling task of a certain target, that is, it is equivalent to executing multiple crawling tasks on a certain target, it is possible to determine whether the error reporting task is successfully executed by checking whether any of the specific execution results contained in the grouped task results is successful, thereby determining whether the error reporting task is successfully executed.

[0107] 207. Based on the grouped task results, determine the task completion rate of each task batch.

[0108] The task completion rate is used to characterize the proportion of successful crawling in the grouped task results in the task batch.

[0109] Based on the aforementioned step 204, it can be known that the method of this embodiment can actually be based on multiple loop executions, that is, after the crawling operation is performed according to the task grouping, the tasks that failed to crawl can be grouped and executed again according to the method of the aforementioned steps. Therefore, in actual applications, as long as there are tasks in any batch that can be successfully completed, it means that there is a crawling failure caused by the failure to open the page normally. Therefore, in this step, the task completion rate of each task batch can be counted as a basis for determining whether it is necessary to continue the loop execution.

[0110] 208. If it is determined that the error reporting task is not executed successfully, the error reporting task is determined as an abnormal task and output.

[0111] Based on the above steps, it can be seen that since the tasks are executed according to task groups, and each group actually corresponds to multiple crawling operations of a task target, therefore, in the process of determining the abnormal task in this step, it can be as follows: if the error reporting task corresponding to the group task fails to execute, then the error reporting task is determined as an abnormal task and output.

[0112] Furthermore, since the method of this embodiment may be executed based on multiple cycles, after the task completion rate is determined in the aforementioned step, the method of determining abnormal tasks in this step needs to consider the task completion rate. Based on this, this step may include:

[0113] On the one hand, if the task completion rate is lower than the preset ratio threshold, the error task is determined as an abnormal task and output. In this case, it means that there are no tasks that failed to crawl due to page opening abnormality, which means that the tasks that have not been successfully crawled so far are indeed tasks that the crawler cannot crawl, so these tasks can be determined as abnormal tasks and output to the user.

[0114] On the other hand, if the task completion rate is higher than the preset ratio threshold, the error reporting task will be added to the task group in the next task batch. In this case, it means that there may be tasks that failed to crawl due to the failure of the page to open normally, so further identification is needed. Specifically, the task can be added to the task group in the next task batch so that the crawling operation can be re-executed later. Similarly, when the preset ratio threshold is 0% and the task completion rate is higher than 0%, it means that the network where the current crawler is located is normal, and you can try to crawl the next batch of tasks again. Otherwise, it means that there may be a network abnormality that causes the failure of crawling of all tasks in this batch, so there is no need to use crawlers to crawl, so as to reduce unnecessary system consumption.

[0115] Furthermore, as a response to the above Figure 1 In order to realize the method shown in the figure, the embodiment of the present invention also provides a crawler task error processing device for Figure 1 The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 3 As shown, the device includes: a grouping unit 31, an execution unit 32, a first judgment unit 33 and a first determination unit 34, wherein

[0116] The grouping unit 31 may be used to group the error reporting tasks according to the task parameters after acquiring the error reporting tasks to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task;

[0117] An execution unit 32, which can be used to execute the error reporting task according to the task grouping obtained by the grouping unit 31 through a crawler, and obtain a grouping task result of each task group;

[0118] A first judging unit 33, which can be used to judge whether the error reporting task is successfully executed according to the grouping task result obtained by the executing unit 32;

[0119] The first determining unit 34 may be configured to determine the error reporting task as an abnormal task and output the abnormal task if the first determining unit 33 determines that the error reporting task is not successfully executed.

[0120] Furthermore, as a response to the above Figure 2 In order to realize the method shown in the figure, the embodiment of the present invention also provides a crawler task error processing device for Figure 2 The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 4 As shown, the device includes: a grouping unit 41, an execution unit 42, a first judgment unit 43 and a first determination unit 44, wherein

[0121] The grouping unit 41 may be used to group the error reporting tasks according to task parameters after acquiring the error reporting tasks to obtain task groups, wherein each task group includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task;

[0122] An execution unit 42, which can be used to execute the error reporting task according to the task grouping obtained by the grouping unit 41 through a crawler, and obtain a grouping task result of each task group;

[0123] A first judging unit 43, which can be used to judge whether the error reporting task is successfully executed according to the grouping task result obtained by the executing unit 42;

[0124] The first determining unit 44 may be configured to determine the error reporting task as an abnormal task and output the abnormal task if the first determining unit 43 determines that the error reporting task is not successfully executed.

[0125] Furthermore, the grouping unit 41 includes:

[0126] The acquisition module 411 may be used to acquire the task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawled page type information, crawling depth information and error information;

[0127] The allocating module 412 may be configured to allocate the plurality of error reporting tasks into a plurality of task groups according to the task parameters acquired by the acquiring module 411 , wherein each of the task groups includes at least one error reporting task having the same task parameters.

[0128] Furthermore, the device also includes:

[0129] The second judging unit 45 may be used to judge whether the number of tasks in each of the task groups obtained by the grouping unit 41 reaches a task threshold;

[0130] The copying unit 46 may be configured to copy the error-reporting tasks in the task group if the second determining unit 45 determines that the number of tasks in each task group does not reach the task threshold, so that the number of tasks reaches the task threshold.

[0131] Furthermore, the execution unit 42 includes:

[0132] The execution module 421 may be used to execute each of the error reporting tasks according to the task grouping through the crawler, and obtain the execution result corresponding to each of the error reporting tasks;

[0133] The statistics module 422 may be used to count the execution results of all the error reporting tasks in the task group obtained by the execution module 421 to obtain the group task results.

[0134] Furthermore, the first determining unit 43 includes:

[0135] The judgment module 431 may be used to judge whether there is at least one execution result that is successfully executed among the multiple execution results corresponding to the grouped task result;

[0136] The first determination module 432 may be configured to determine that the error reporting task corresponding to the grouping task is successfully executed if the determination module 431 determines that at least one of the multiple execution results corresponding to the grouping task result is successfully executed;

[0137] The second determination module 433 may be configured to determine that the error reporting task corresponding to the grouping task has failed to execute if the determination module 431 determines that there is not at least one execution result that is successfully executed among the multiple execution results corresponding to the grouping task result;

[0138] The first determining unit 44 may be specifically configured to determine and output the error reporting task as an abnormal task if the error reporting task corresponding to the grouping task fails to execute.

[0139] Furthermore, the device also includes:

[0140] A setting unit 47, which can be used to set the task groups obtained by the multiple grouping units 41 as a task batch;

[0141] The device also includes:

[0142] The second determining unit 48 may be used to determine the task completion rate of each task batch set by the setting unit 47 according to the grouped task results obtained by the executing unit 42, and the task completion rate may be used to characterize the proportion of successful crawling in the grouped task results in the task batch;

[0143] The first determining unit 44 includes:

[0144] The first determination module 441 may be configured to determine the error-reporting task as an abnormal task and output the task if the task completion rate is lower than a preset ratio threshold;

[0145] The adding module 442 may be configured to add the error reporting task to the task group in the next task batch if the task completion rate is higher than a preset ratio threshold.

[0146] By means of the above technical solution, the embodiment of the present invention provides a method and device for processing crawler task errors.

[0147] In order to solve the problem of high labor cost in the process of handling the existing crawler task error, the present invention obtains the error reporting task by grouping the error reporting task according to the task parameters after obtaining the error reporting task, obtains the task group, and then executes the error reporting task according to the task group by the crawler, and obtains the group task result of each task group, and then judges whether the error reporting task is successfully executed according to the group task result, and finally, if it is unsuccessful, the error reporting task is determined as an abnormal task and output, thereby completing the processing of the crawler task error. In the above scheme, each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks the crawling failure after the crawler executes the crawling task. This makes it possible to automatically obtain the previous crawling failed task when processing the crawler task error, and execute the error reporting task again in the form of grouping based on the task group, so that it can be ensured that the failed error reporting task can be automatically obtained and executed again after the crawler crawls and fails, and the error reporting task caused by the failure of the network or page opening can be avoided. It can be executed again, and there is no need to manually judge whether the crawling failure is caused by the above problem, saving labor costs. At the same time, after executing according to task groups, when it is determined that the error-reporting task still cannot be crawled successfully, the error-reporting task is identified as an abnormal task and output, which can ensure that the user can directly determine that the task is indeed a task that cannot be crawled based on the output results without manual identification, further saving manpower and thus solving the problem of excessively high labor costs.

[0148] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and a method for processing crawler task errors that can be automatically executed is implemented by adjusting kernel parameters, so that the crawler task error processing process does not require manual operation, solving the problem of high labor costs in the prior art.

[0149] An embodiment of the present invention provides a storage medium on which a program is stored. When the program is executed by a processor, the method for processing crawler task errors is implemented.

[0150] An embodiment of the present invention provides a processor, which is used to run a program, wherein the method for processing crawler task errors is executed when the program is running.

[0151] The embodiment of the present invention provides a device 50, such as Figure 5 As shown, the device includes at least one processor 501, and at least one memory 502 and a bus 503 connected to the processor; wherein the processor 501 and the memory 502 communicate with each other through the bus 503; the processor 501 is used to call the program instructions in the memory to execute the above-mentioned crawler task error processing method.

[0152] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0153] The present application also provides a computer program product, which, when executed on a process management device, is suitable for executing a program that is initialized with the following method steps: after obtaining an error reporting task, grouping the error reporting task according to task parameters to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task; executing the error reporting task according to the task group by the crawler, and obtaining the grouping task result of each task group; judging whether the error reporting task is executed successfully based on the grouping task result; if unsuccessful, determining the error reporting task as an abnormal task and outputting it.

[0154] Furthermore, after obtaining the error reporting tasks, the error reporting tasks are grouped according to the task parameters, and the obtained task groups include:

[0155] Acquire task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawling page type information, crawling depth information and error information;

[0156] According to the task parameters, a plurality of the error reporting tasks are allocated into a plurality of the task groups, wherein each of the task groups includes at least one error reporting task having the same task parameters.

[0157] Furthermore, before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes:

[0158] Determine whether the number of tasks in each of the task groups reaches a task threshold;

[0159] If not reached, a copy operation is performed on the error-reporting tasks in the task group so that the number of tasks reaches the task threshold.

[0160] Furthermore, executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group includes:

[0161] Executing each of the error reporting tasks according to the task groups by the crawler, and obtaining the execution result corresponding to each of the error reporting tasks;

[0162] The execution results of all the error reporting tasks in the task group are counted to obtain the group task result.

[0163] Further, judging whether the error reporting task is successfully executed according to the grouping task result includes:

[0164] Determine whether at least one of the plurality of execution results corresponding to the grouped task result is successfully executed;

[0165] If so, determining that the error reporting task corresponding to the grouping task is executed successfully;

[0166] If it does not exist, determining that the error reporting task corresponding to the grouping task fails to execute;

[0167] If the task fails, the error reporting task is determined as an abnormal task and outputted, including:

[0168] If the error reporting task corresponding to the grouping task fails to execute, the error reporting task is determined as an abnormal task and output.

[0169] Furthermore, before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes:

[0170] Grouping a plurality of the tasks into a task batch;

[0171] Before determining the error reporting task as an abnormal task and outputting it, the method further includes:

[0172] Determine the task completion rate of each task batch according to the grouped task results, wherein the task completion rate is used to characterize the proportion of successful crawling in the grouped task results in the task batch;

[0173] The step of determining the error reporting task as an abnormal task and outputting the task includes:

[0174] If the task completion rate is lower than a preset ratio threshold, the error-reporting task is determined as an abnormal task and output;

[0175] If the task completion rate is higher than a preset ratio threshold, the error reporting task is added to the task group in the next task batch.

[0176] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable process management device to generate a machine, so that the instructions executed by the processor of the computer or other programmable process management device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0178] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0179] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0182] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing crawler task errors, characterized in that: include: After obtaining the error reporting tasks, the error reporting tasks are grouped according to the task parameters to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task that feedbacks crawling failure after the crawler executes the crawling task; Execute the error reporting task according to the task grouping by a crawler, and obtain the grouping task result of each task group; According to the grouping task result, judging whether the error reporting task is executed successfully; If unsuccessful, the error reporting task is determined as an abnormal task and output; Before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes: Grouping a plurality of the tasks into a task batch; Before determining the error reporting task as an abnormal task and outputting it, the method further includes: Determine the task completion rate of each task batch according to the grouped task results, wherein the task completion rate is used to characterize the proportion of successful crawling in the grouped task results in the task batch; The step of determining the error reporting task as an abnormal task and outputting the task includes: If the task completion rate is lower than a preset ratio threshold, the error-reporting task is determined as an abnormal task and output; If the task completion rate is higher than a preset ratio threshold, the error reporting task is added to the task group in the next task batch.

2. The method according to claim 1, characterized in that After obtaining the error reporting tasks, the error reporting tasks are grouped according to the task parameters, and the obtained task groups include: Acquire task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawling page type information, crawling depth information and error information; According to the task parameters, the plurality of error reporting tasks are allocated into a plurality of task groups, wherein each of the task groups includes at least one error reporting task having the same task parameters.

3. The method according to claim 2, characterized in that Before executing the error reporting task according to the task group by the crawler and obtaining the group task result of each task group, the method further includes: Determine whether the number of tasks in each of the task groups reaches a task threshold; If not reached, a copy operation is performed on the error-reporting tasks in the task group so that the number of tasks reaches the task threshold.

4. The method according to claim 3, characterized in that The crawler executes the error reporting task according to the task group and obtains the group task result of each task group, including: Executing each of the error reporting tasks according to the task groups by the crawler, and obtaining the execution result corresponding to each of the error reporting tasks; The execution results of all the error reporting tasks in the task group are counted to obtain the group task result.

5. The method according to claim 4, characterized in that The determining, according to the grouping task result, whether the error reporting task is successfully executed comprises: Determine whether at least one of the plurality of execution results corresponding to the grouped task result is successfully executed; If so, determining that the error reporting task corresponding to the grouping task is executed successfully; If it does not exist, determining that the error reporting task corresponding to the grouping task fails to execute; If the task fails, the error reporting task is determined as an abnormal task and outputted, including: If the error reporting task corresponding to the grouping task fails to execute, the error reporting task is determined as an abnormal task and output.

6. A crawler task error processing device, characterized in that: include: A grouping unit, for grouping the error reporting tasks according to task parameters after acquiring the error reporting tasks, to obtain task groups, wherein each of the task groups includes at least one error reporting task, and the error reporting task is a task for which the crawler reports a crawling failure after executing the crawling task; An execution unit, used for executing the error reporting task according to the task group through a crawler, and obtaining a group task result of each task group; A first judging unit, configured to judge whether the error reporting task is successfully executed according to the grouping task result; A first determining unit, configured to determine the error reporting task as an abnormal task and output the abnormal task if it is determined that the error reporting task is not successfully executed; A setting unit, used for setting the task groups obtained by the plurality of grouping units into a task batch; The device also includes: A second determining unit is used to determine the task completion rate of each task batch set by the setting unit according to the grouped task results obtained by the executing unit, wherein the task completion rate can be used to characterize the proportion of successful crawling in the grouped task results in the task batch; The first determining unit includes: A first determination module may be used to determine the error-reporting task as an abnormal task and output it if the task completion rate is lower than a preset ratio threshold; The adding module can be used to add the error reporting task to the task group in the next task batch if the task completion rate is higher than a preset ratio threshold.

7. The device according to claim 6, characterized in that The grouping unit comprises: An acquisition module, used for acquiring task parameters of the error reporting task, wherein the task parameters include at least one of uniform resource locator information, crawling page type information, crawling depth information and error information; An allocating module is used to allocate the plurality of error reporting tasks into a plurality of task groups according to the task parameters, wherein each of the task groups contains at least one error reporting task with the same task parameters.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the crawler task error processing method described in any one of claims 1 to 5.

9. A device, characterized in that: The device includes at least one processor, and at least one memory and bus connected to the processor; wherein the processor and memory communicate with each other through the bus; the processor is used to call the program instructions in the memory to execute the crawler task error processing method as described in any one of claims 1 to 5.

Images