Downloading task processing method and device, computer equipment and storage medium

Abstract

The invention relates to a downloading task processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a downloading request through an access service, constructing a corresponding target downloading task for the downloading request, and adding a corresponding task identifier to a task queue; the task identification is taken out from the task queue through the scheduling service, then the task identification is reported to the task state management service for task activation, and the task identification is scheduled to the downloading service and the analysis service; downloading data according to a target downloading task corresponding to the task identifier through a downloading service, obtaining and analyzing corresponding downloading data according to the task identifier through an analysis service, and reporting a task state to a task state management service; and when the task state management service monitors the downloading task of which the corresponding task state meets the retry condition, adding the task identifier of the downloading task meeting the retry condition to the task queue again. By adopting the method, the reliability of processing the downloading task can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for processing a download task. Background Art

[0002] With the rapid development of Internet technology, the amount of information contained in the Internet is growing rapidly. People need to spend a lot of energy to find the information they need in the massive amount of information. Therefore, people have an increasing demand for obtaining the information they need anytime and anywhere.

[0003] Data can be collected by using web crawler technology. However, since crawling needs are mostly temporary and urgent, technicians often quickly develop a single-function system based on an open source framework for a specific crawling need. Due to the lack of system-level considerations and designs, it is difficult to have high reliability, scalability and maintainability. Even for each crawling need, it is necessary to repeatedly develop a corresponding crawling system. The development speed is slow and the general applicability is low, resulting in a waste of manpower and equipment resources, and it can no longer meet the demand for such large amounts of data. Summary of the invention

[0004] Based on this, it is necessary to provide a download task processing method, device, computer equipment, computer-readable storage medium and computer program product to address the above-mentioned technical problems. The method provides a platform-level, general, business-independent download task processing process that can improve the reliability of download task processing and reduce the manpower cost of continuous development and operation.

[0005] In a first aspect, the present application provides a method for processing a download task. The method comprises:

[0006] Receive a download request through an access service, construct a corresponding target download task for the download request, and add a task identifier of the target download task to a task queue;

[0007] After taking out the task identifier from the task queue through the scheduling service, the taken out task identifier is reported to the task status management service for task activation, and the taken out task identifier is scheduled to the download service and the parsing service;

[0008] Download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service;

[0009] The task status management service is used to monitor the activated download tasks. When a download task whose corresponding task status satisfies the retry condition is detected, the task identifier of the download task that satisfies the retry condition is added back to the task queue.

[0010] In a second aspect, the present application also provides a download task processing device. The device comprises:

[0011] A task access module, used to receive a download request through an access service, construct a corresponding target download task for the download request, and add a task identifier of the target download task to a task queue;

[0012] A task scheduling module, used for taking out a task identifier from the task queue through a scheduling service, reporting the taken out task identifier to a task status management service for task activation, and scheduling the taken out task identifier to a download service and a parsing service;

[0013] The task processing module is used to download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service;

[0014] The status monitoring module is used to monitor the activated download tasks through the task status management service. When a download task whose corresponding task status meets the retry condition is monitored, the task identifier of the download task that meets the retry condition is added back to the task queue.

[0015] In one embodiment, the task access module is also used to receive a download request through an access service, wherein the download request carries download parameters, and the download parameters include a download keyword and an access interface identifier for accessing a site; generate a corresponding task identifier for the download request through the access service, and encapsulate the task according to the task identifier and the download parameters to obtain a structured task.

[0016] In one embodiment, the task access module is also used to generate a task identifier for the download request through an access service; when it is determined that the multiple download requests are for the same access site based on the respective download parameters in the multiple download requests, the same task group identifier is assigned to the multiple download requests; for each download request in the multiple download requests, task encapsulation is performed based on the task group identifier, the task identifier and the download parameters to obtain multiple structured tasks belonging to the same task group.

[0017] In one embodiment, the task access module is also used to check the download keywords corresponding to multiple structured tasks belonging to the same task group; if there are at least two structured tasks with similar corresponding download keywords among the multiple structured tasks belonging to the same task group, the at least two structured tasks are deduplicated.

[0018] In one embodiment, the task access module is also used to register the task with the task status management service through the access service according to the task identifier of the target download task before adding the task identifier of the target download task to the task queue, so as to instruct the task status management service to persist the target download task.

[0019] In one embodiment, the download request carries a request object identifier and a request priority; the task access module is also used to determine multiple task queues corresponding to the request object identifier; determine a target queue that matches the request priority from the multiple task queues; and add the task identifier of the target download task to the target queue.

[0020] In one embodiment, the task processing module is also used to download data according to the target download task corresponding to the task identifier through the download service, add the data index of the downloaded data to the download data queue, and report the task status to the task status management service; retrieve the data index from the download data queue through the scheduling service, and schedule the retrieved data index to the parsing service; obtain and parse the corresponding download data according to the data index through the parsing service, add the result index of the parsing result to the parsing result queue, and report the task status to the task status management service.

[0021] In one embodiment, the task processing module is also used to download data through the download service according to the target download task corresponding to the task identifier; in the case of successful download, a data index is generated for the downloaded data and associated with the task identifier, the generated data index is added to the download data queue, and a message indicating that the task download corresponding to the task identifier is successfully downloaded is reported to the task status management service; in the case of failed download, a message indicating that the task download corresponding to the task identifier is failed is reported to the task status management service.

[0022] In one embodiment, the task processing module is also used to obtain and parse the corresponding downloaded data according to the data index through the resolution service; in the case of successful resolution, a result index is generated for the resolution result and associated with the task identifier, the generated result index is added to the resolution result queue, and a message indicating successful task resolution corresponding to the task identifier is reported to the task status management service; in the case of failed resolution, a message indicating failed task resolution corresponding to the task identifier is reported to the task status management service.

[0023] In one embodiment, the task processing module is also used to construct a download request according to the access interface identifier of a secondary access site belonging to the visited site when the parsing result includes an access interface identifier of the secondary access site, and send the constructed download request to the access service.

[0024] In one embodiment, the apparatus further comprises:

[0025] A frequency control module, configured to obtain a request rate interval for the access site corresponding to the download request through a frequency control service; determine a task execution success rate for the access site according to the task status of the historical download tasks involving the access site reported by the download service and the resolution service through the frequency control service, and adjust the request rate for the access site within the request rate interval according to the task execution success rate;

[0026] The task scheduling module is further used to query the request rate for the access site from the frequency control service, and schedule the retrieved task identifier to the download service according to the queried request rate.

[0027] In one embodiment, the scheduling module is also used to query the download parameters of the target download task corresponding to the retrieved task identifier; if there is a historical download task whose corresponding download parameters are similar to the download parameters of the target download task, query whether there is a cached parsing result corresponding to the historical download task; if there is a cached parsing result corresponding to the historical download task, index the result of the cached parsing result and associate it with the task identifier of the target download task, add the result index to the parsing result queue, and report a message that the task parsing is successful corresponding to the task identifier to the task status management service; if there is no cached parsing result corresponding to the historical task, report the retrieved task identifier to the task status management service for task activation, and schedule the retrieved task identifier to the download service and the parsing service.

[0028] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the processing method of the download task are implemented.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned download task processing method when executed by a processor.

[0030] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the processing method of the above-mentioned download task are implemented.

[0031] The above-mentioned download task processing method, device, computer equipment, storage medium and computer program product receive download requests through access services, construct corresponding target download tasks for download requests, add the task identifier of the target download task to the task queue, and make the entire task processing engine highly scalable through access services, so as to efficiently access a large number of crawling requirements; after taking out the task identifier from the task queue through the scheduling service, report the taken out task identifier to the task status management service for task activation, and schedule the taken out task identifier to the download service and the parsing service; download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download task according to the task identifier through the parsing service Download data and report the task status to the task status management service. Based on this, the processing efficiency of the download task can be improved, and the robustness of the entire task processing engine can be ensured by dividing each function based on the data flow. The throughput of the entire task processing engine can be improved through task encapsulation, scheduling services, download services, and parsing services. In addition, through the task status management service, the download tasks that have been activated are monitored. When the download tasks whose corresponding task status meets the retry conditions are monitored, the task identifier of the download tasks that meet the retry conditions is added back to the task queue. Based on this, the execution status of the download tasks can be grasped in real time through task status monitoring. The retry mechanism based on task status can ensure the integrity and success rate of the download tasks, making the task processing engine more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 An application environment diagram of a method for processing a download task in an embodiment;

[0033] Figure 2 A flowchart of a method for processing a download task in another embodiment;

[0034] Figure 3A flowchart of a method for processing a download task in one embodiment;

[0035] Figure 4 is a schematic diagram of multiple structured tasks of the same task group in one embodiment;

[0036] Figure 5 A functional diagram of an access service in an embodiment;

[0037] Figure 6 A schematic diagram of a process of processing a download request through an access service in one embodiment;

[0038] Figure 7 A functional schematic diagram of a scheduling service in one embodiment;

[0039] Figure 8 A functional schematic diagram of a download service in one embodiment;

[0040] Fig. 9 A functional schematic diagram of a parsing service in one embodiment;

[0041] Fig.10 A schematic diagram of a task scheduling process in one embodiment;

[0042] Fig.11 A functional diagram of a task status management service in one embodiment;

[0043] Fig.12 It is a schematic diagram of the relationship between request rate, success rate, and success rate (i.e., task execution success rate);

[0044] Fig.13 A functional schematic diagram of a frequency control service in one embodiment;

[0045] Fig.14 A functional schematic diagram of a cache service in one embodiment;

[0046] Fig.15 A flowchart of a method for processing a download task in a detailed embodiment;

[0047] Fig.16 A schematic diagram of the system structure of a task processing engine in one embodiment;

[0048] Fig.17 is a schematic diagram of the scheduling relationship between various services in one embodiment;

[0049] Fig.18 A comparative diagram of the performance improvement of the task processing engine;

[0050] Fig.19 is a structural block diagram of a processing device for downloading tasks in one embodiment;

[0051] Fig. 20 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The download task processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. In one embodiment, the server 104 can receive a download request through an access service, construct a corresponding target download task for the download request, add the task identifier of the target download task to the task queue, and report the task identifier taken out from the task queue to the task status management service for task activation after taking out the task identifier through the scheduling service, and schedule the taken out task identifier to the download service and the parsing service, and download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, report the task status to the task status management service, and monitor the download task that has been activated through the task status management service. When the download task whose corresponding task status meets the retry condition is monitored, the task identifier of the download task that meets the retry condition is added to the task queue again.

[0054] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0055] The terminal 102 may be a terminal used by business personnel, through which business personnel can initiate crawling requirements, i.e., download requests, to the task processing engine. The server 104 is a general term for servers that provide all background services for the task processing engine. The server may be a server cluster composed of multiple distributed services. It is understood that the computer programs that implement the above services may run on different physical devices.

[0056] like Figure 1As shown, the server 104 may include a server 1041 providing access services, a server providing scheduling services, the scheduling services may access the task queue MQ1, a server providing download services, the download services may access the download data queue MQ2, and a server providing parsing services, the parsing services may access the parsing result queue MQ3. Optionally, these queues are message queues that can implement asynchronous processing, such as ActiveMQ, RabbitMQ, Kafka, etc. The server 104 also includes a server 1045 providing task status management services, the task status management services may access a database DB for storing tasks and task information (e.g., task identification, task status, task download parameters, task request objects, task access regions, etc.), the database may be implemented based on MongoDB (a database based on distributed file storage), and the task status management services may also access the task queue MQ1.

[0057] Reference Figure 1 The server providing access service can communicate with the server providing scheduling service through the network. The server providing scheduling service can also communicate with the server providing download service, the server providing resolution service and the server providing task status management service through the network. The server providing download service and the server providing resolution service can communicate with the server providing task status management service through the network.

[0058] Optionally, server 104 also includes a server providing a cache service and a server providing a frequency control service. The server providing the cache service can access the resolution result queue MQ3 to cache the resolution results. The server providing the cache service can communicate with the servers providing the scheduling service and the resolution service through the network. The server providing the frequency control service can communicate with the servers providing the access service, the scheduling service, the download service, and the resolution service through the network.

[0059] like Figure 2 FIG. 1 is a schematic diagram of an application environment of a method for processing a download task in a detailed embodiment. Figure 2 In order to improve the availability of the task processing engine, it can be deployed in a two-site multi-center manner. Figure 2As shown, the first region and the second region are deployed respectively. Among them, each service is deployed independently in each region. Access service, scheduling service, download service and analysis service are deployed on the server in the first region, and are only called within the first region. Access service, scheduling service, download service and analysis service and other services can be deployed on the server in the second region, and are only called within the second region. Various queues can be implemented based on Kafka. Since Kafka does not yet support remote disaster recovery and remote access, task queues, download data queues and analysis result queues are deployed in each region. Task persistence and task status monitoring can be implemented based on MongoDB, which supports remote disaster recovery and can be called from different regions. A task status management service can be deployed in two regions.

[0060] In one embodiment, Figure 3 As shown, a method for processing a download task is provided, and the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0061] Step 302: Receive a download request through the access service, construct a corresponding target download task for the download request, and add the task identifier of the target download task to the task queue.

[0062] Among them, the access service is used to process the download request initiated by the request object, and the request object can be a user with crawling requirements, such as business personnel, developers, etc. In the embodiment of the present application, the access service is the entrance to the entire task processing engine. It provides a unified access interface for a large number of download requirements. When the request object has a crawling requirement for the target site, the download parameters can be set according to the crawling requirements, and the access interface can be accessed according to the download parameters to initiate a download request to the access service. The download task is a structured task constructed in a unified format based on the download request. The download task is the basis for the versatility and scalability of the entire task processing engine.

[0063] The task identifier is used to identify the download task. Through the access service, when a download task is generated, a corresponding task identifier can be generated for the download request. The task identifier is global and can realize the full monitoring and management of the identified download task.

[0064] In one embodiment, a download request is received through an access service, and a corresponding target download task is constructed for the download request, including: receiving a download request through an access service, the download request carries download parameters, the download parameters include a download keyword and an access interface identifier for accessing a site; generating a corresponding task identifier for the download request through the access service, and encapsulating the task according to the task identifier and the download parameters to obtain a structured task.

[0065] Among them, the download keyword is a keyword used for data extraction, for example, the keyword can be the title of a song, or the name of a singer, etc. The access interface identifier of the access site is the resource data that can access a certain site, such as the access address or link of a certain page of an APP, the access address of a certain web page, and so on. Specifically, the server can construct a legal web page access request suitable for the access site according to the construction logic of the web page request of the access site through the access service, that is, http request according to the download parameters, and then generate a corresponding task identifier for the web page access request, perform task encapsulation on the task identifier and the web page access request, and obtain a structured download task. Optionally, other control parameters can also be added during encapsulation. For example, other control parameters can be the type of data to be captured, the frequency of access, and so on.

[0066] In one embodiment, a corresponding task identifier is generated for a download request through an access service, and tasks are encapsulated according to the task identifier and download parameters to obtain a structured task, including: generating a task identifier for a download request through an access service; assigning the same task group identifier to multiple download requests when multiple download requests are determined to be for the same access site according to the respective download parameters in the multiple download requests; for each of the multiple download requests, tasks are encapsulated according to the task group identifier, the task identifier and the download parameters to obtain multiple structured tasks belonging to the same task group. In this way, multiple structured tasks (Tasks) can be combined into a task group (Job) according to the needs of the request object, so that a parsing result of a bounded task group can be obtained according to the one task group, for consumption by the request object that initiated the multiple download requests.

[0067] like Figure 4 FIG. 1 is a schematic diagram of multiple structured tasks of the same task group in one embodiment. Figure 4 The global identifier of each structured task may include a task group identifier (JobId) and a task identifier (TaskId). Multiple structured tasks in the same task group have the same task group identifier (JobId) and different task identifiers (TaskId). The structured task is constructed according to a web page access request, i.e., an HTTP Request. The structured task may also include control parameters corresponding to the download task, custom data, and the like.

[0068] In one embodiment, the request object can also register its own object identifier with the access service of the task processing engine. Through the object identifier, the access service can perform access control, that is, only the registered object identifier can access the access interface, and can also implement access control policies, access rights dynamic management policies, etc. based on the object identifier. In this embodiment, through access control, it is possible to prevent the task processing engine from being accessed by a large number of malicious requests and affecting the processing efficiency of the task processing engine for normal requests, and to ensure the overall performance of the task processing engine.

[0069] In one embodiment, the server can also check the download keywords corresponding to multiple structured tasks belonging to the same task group through the access service; if there are at least two structured tasks with similar corresponding download keywords among the multiple structured tasks belonging to the same task group, the at least two structured tasks are deduplicated. In this embodiment, by filtering the requests with similar download keywords in the same group of tasks, it is possible to prevent repeated crawling and avoid the formation of crawling loops in the multi-level crawling process, which causes the task processing engine to repeatedly process a large number of download tasks and affects the overall performance of the task processing engine.

[0070] like Figure 5 FIG. 1 is a functional diagram of an access service in an embodiment. Figure 5 , the access service can provide the following functions, but is not limited to these:

[0071] 1. Provide a unified access interface, that is, the access service can shield the implementation details and internal call relationships of the entire task processing engine, and provide a concise and feature-rich API call interface;

[0072] 2. Construct a legal web page request, that is, the access service maintains the corresponding network request construction logic for all accessible sites, and can construct a legal web page access request suitable for the access site based on the download parameters, completing the conversion from download parameters to network requests;

[0073] 3. Task encapsulation, that is, the access service generates a global task identifier for the download task based on the access interface, and can also attach control parameters to the task. According to the global task identifier, the constructed web page access request and the control parameters, a standard structured task entity is constructed, thereby realizing the task encapsulation of the download request;

[0074] 4. Access control, that is, the access service can control access through object identification, and only registered objects can access authorized interfaces;

[0075] 5. Duplicate request filtering, that is, the access service can filter duplicate requests in the same set of tasks to prevent duplicate crawling.

[0076] In one embodiment, before adding the task identifier of the target download task to the task queue, the method further includes: registering the task with the task status management service according to the task identifier of the target download task through the access service to instruct the task status management service to persist the target download task, and after receiving a notification of successful persistence, executing the step of adding the task identifier of the target download task to the task queue. Persistence is the process in which the server registers the task with the task status management service according to the task identifier of the download task and the download task through the access service, so that the task status management service records the task information about the download task in the database, such as the task identifier, task status, download parameters of the task, etc.

[0077] In this embodiment, before the task identifier of the download task is added to the task queue, it is necessary to go through the task persistence step, that is, initiate task registration to the task status management service to complete the persistence of the download task. Only when the download task is persisted successfully, will the corresponding task identifier be written into the task queue to wait for the subsequent processing steps, and will the information that the task has been successfully connected be returned to the request object, which can ensure the reliable reception of the download request of the request object and the reliability of the task processing engine.

[0078] Optionally, before the download task is queued, it can also go through some steps, such as judging whether the task queue is full, etc. That is to say, after the preceding steps of judging whether the task queue is full, generating the download task, judging whether it has been persisted, etc. are successfully executed, the corresponding task identifier will be written into the task queue, and the information that the task has been successfully connected will be returned to the request object.

[0079] Optionally, the task queue can be determined by user or request priority. The download request carries a request object identifier and a request priority; adding the task identifier of the target download task to the task queue includes: determining multiple task queues corresponding to the request object identifier; determining a target queue matching the request priority from the multiple task queues; and adding the task identifier of the target download task to the target queue.

[0080] In this embodiment, in order to achieve the isolation of download tasks between different request objects, each request object corresponds to a set of task queues that belong to it alone. In order to achieve the task scheduling based on priority of a single request object, the task queue corresponding to each request object is a multi-level task queue divided by priority. In terms of scheduling logic, the download task of each request object has a fair opportunity to be scheduled by the scheduling service, and for the download tasks of different priorities of the same request object, they have the opportunity to be scheduled by the scheduling service that matches their priority.

[0081] like Figure 6FIG. 1 is a flow chart of processing a download request through an access service in one embodiment. Figure 6 The flowchart shows the main data flow for processing download requests. First, the download request is received, and then it is determined whether the task queue is full. If the task queue is full, the server directly ends the process through the access service and returns a message that the task processing failed to the request object. If the task queue is not full, the server continues to generate a web page access request according to the download parameters of the download request through the access service, constructs a download task, and registers the download task with the task status management service to complete the persistence of the content and status of the download task. If the persistence fails, the server directly ends the process through the access service and returns a message that the task processing failed to the request object. If the persistence is successful, the corresponding task identifier is written into the task queue and the information that the task has been successfully accessed is returned to the request object.

[0082] Step 304: After taking out the task identifier from the task queue through the scheduling service, the taken out task identifier is reported to the task status management service for task activation, and the taken out task identifier is scheduled to the download service and the parsing service.

[0083] Among them, the scheduling service is used to implement the scheduling of download tasks. The server can monitor the task queue through the scheduling service. When there is a task identifier in the task queue, the task identifier is taken out from the task queue, and the task identifier is scheduled according to a certain scheduling strategy, so that the subsequent download service and parsing service can process the corresponding download task. As the center of the entire task processing engine, the scheduling service can interact with other services. For example, the scheduling service can accept task retry requests from the task status management service and perform secondary scheduling on the corresponding tasks. The scheduling service can also report the taken out task identifier to the task status management service for task activation before scheduling the task, so that the task status management service records that the current task identifier is in an activated state, and then schedules the task.

[0084] like Figure 7 FIG. 1 is a functional diagram of a scheduling service in an embodiment. Figure 7 The scheduling service can realize global task scheduling, downstream service calls, download task isolation of different request objects, task status reporting, etc. The server can execute the above process and realize the above functions through the scheduling service.

[0085] Step 306, download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service.

[0086] Among them, the download service is used to download data according to the target download task corresponding to the task identifier, and add the data index of the downloaded data to the download data queue. Optionally, the download service can be deployed in a distributed manner, and the download tasks from the scheduling service are downloaded in a distributed and parallel manner, which can improve the download efficiency when facing a large number of download tasks. Optionally, the download service can also perform timely download retries for download failures such as network request timeouts and abnormal return codes. Optionally, for download tasks where the download data type is a file, the download service can support file downloads, and perform non-repetitive cloud storage based on the hash value of the file (such as the MD5 value), and write the file's existing address as a data index to the download data queue for processing by the downstream parsing service. Optionally, the download service can also be deployed independently in a network isolation environment, and communicate with other services through limited ports, so that it can be safely open to the external network. In this way, even if the download service uses a variety of dynamic proxy addresses provided by the outside, based on these proxy addresses for a variety of web pages, the network isolation environment can reduce network security risks and ensure the network security of the entire task processing engine.

[0087] like Figure 8 FIG. 1 is a functional diagram of a download service in an embodiment. Figure 8 The download service executes the download task through a dynamic proxy address, and the downloaded data is HTML code and JSON data or a file. If it is a file, the hash value of the file is added to the message queue as a data index. It also supports deduplication of downloaded files to save storage space.

[0088] The parsing service is used to parse the downloaded data from various web pages, extract the information therein, and convert it into data in a fixed format, namely the parsing result, and add the result index of the parsing result to the parsing result queue. Fig. 9 FIG. 1 is a functional diagram of a parsing service in an embodiment. Fig. 9 ,The resolution service can also generate a secondary download request and send it to the access service to construct a new download task.

[0089] The task status management service is used to manage and monitor the task status of tasks. The access service, scheduling service, download service, and parsing service can all report the task status to the task status management service. For example, the access service can register the task with the task status management service based on the task identifier, which is used to notify the task status management service to persist the download task. After the scheduling service takes out the task identifier from the task queue, it reports the taken out task identifier to the task status management service to activate the task, which is actually a report of the task status, used to inform the task status management service that the download task will be scheduled to the download service and the parsing service. The download service can report the corresponding task status to the task status management service based on whether the download task corresponding to the task identifier is downloaded successfully or not. The parsing service can report the corresponding task status to the task status management service based on whether the parsing of the download data corresponding to the task identifier is successful or not. In this way, the status monitoring of the download task in each processing link can be realized, and the reliability of the task processing engine can be improved. The server can improve the versatility, scalability, and reliability of the entire task processing engine by deploying the above-mentioned various services.

[0090] In one embodiment, data is downloaded by a download service according to a target download task corresponding to a task identifier, and the task status is reported to a task status management service. The corresponding download data is obtained and parsed by a parsing service according to the task identifier, and the task status is reported to the task status management service, including: data is downloaded by a download service according to a target download task corresponding to the task identifier, a data index of the downloaded data is added to a download data queue, and the task status is reported to the task status management service; a data index is retrieved from the download data queue by a scheduling service, and the retrieved data index is scheduled to a parsing service; a corresponding download data is obtained and parsed by a parsing service according to the data index, a result index of the parsing result is added to a parsing result queue, and the task status is reported to the task status management service.

[0091] In one embodiment, after the corresponding download data is acquired and parsed according to the data index by the parsing service, and the result index of the parsing result is added to the parsing result queue, the method further includes: when the parsing result includes the access interface identifier of the secondary access site belonging to the visited site, a download request is constructed according to the access interface identifier of the secondary access site, and the constructed download request is sent to the access service. The access site in the download parameters corresponding to the download task can be regarded as the primary access site. By downloading and parsing the data of the download task, the access site included in the parsing result is the secondary access site. When the parsing result includes the interface identifier of such a secondary access site, the server can construct a download request according to the interface identifier of the secondary access site through the parsing service, and send the constructed download request to the access service to realize the secondary data download of the original access site.

[0092] In one embodiment, downloading data according to a target download task corresponding to a task identifier is performed through a download service, a data index of the downloaded data is added to a download data queue, and the task status is reported to a task status management service, including: downloading data according to a target download task corresponding to the task identifier is performed through a download service; when the download is successful, a data index is generated for the downloaded data and associated with the task identifier, the generated data index is added to the download data queue, and a message that the task corresponding to the task identifier is successfully downloaded is reported to the task status management service; the method also includes:

[0093] In the event of a download failure, a message indicating that the task download failure corresponding to the task identifier has occurred is reported to the task status management service.

[0094] In one embodiment, the corresponding downloaded data is obtained and parsed according to the data index through the parsing service, the result index of the parsing result is added to the parsing result queue, and the task status is reported to the task status management service, including: obtaining and parsing the corresponding downloaded data according to the data index through the parsing service; in the case of successful parsing, generating a result index for the parsing result and associating it with the task identifier, adding the generated result index to the parsing result queue, and reporting a message that the task parsing corresponding to the task identifier is successful to the task status management service; the method also includes: in the case of failed parsing, reporting a message that the task parsing corresponding to the task identifier is unsuccessful to the task status management service.

[0095] like Fig.10 FIG. 1 is a flowchart of a task scheduling process in an embodiment. Fig.10 , monitor whether the task queue is empty through the scheduling service. If the task queue is empty, continue to monitor. If it is not empty, take out the task identifier from the task queue and schedule it to the download service. The download service downloads data for the corresponding download task. If the data download fails, the task failure is fed back to the task status management service. If the data download is successful, the task download success is fed back to the task status management service, and the data index of the downloaded data is added to the download data queue. The parsing service can perform data parsing. If the data parsing fails, the task failure is fed back to the task status management service. If the data parsing is successful, the task success is fed back to the task status management service, and the parsing result is added to the parsing result queue.

[0096] Step 308, monitoring the activated download tasks through the task status management service, and when monitoring the download tasks whose corresponding task status meets the retry condition, adding the task identifier of the download tasks meeting the retry condition to the task queue again.

[0097] The task status management service can monitor the task status of each download task. For a download task that has been scheduled, that is, a download task that has been activated, if its task status shows failure, it may be due to a network problem that caused the download to fail or the parsing to fail. Such a download task can be rescheduled. If the number of rescheduling is less than the set number, that is, the retry condition is met, the task identifier of the download task that meets the retry condition will be added to the task queue again and rescheduled. This retry mechanism based on task status can ensure the integrity and success rate of the download task, making the task processing engine more reliable. Setting the retry condition can also prevent unlimited retries from affecting the scheduling and processing of other download tasks by the task processing engine.

[0098] like Fig.11 FIG. 1 is a functional diagram of a task status management service in an embodiment. Fig.11 , the access service can register the task with the task status management service based on the task identifier, which is used to notify the task status management service to persist the download task. After taking out the task identifier from the task queue, the scheduling service reports the taken out task identifier to the task status management service to activate the task. The download service can report the corresponding task status to the task status management service based on whether the download of the download task corresponding to the task identifier is successful or not. The parsing service can report the corresponding task status to the task status management service based on whether the parsing of the download data corresponding to the task identifier is successful or not. The recorded task status can be used to filter out the download tasks that meet the retry conditions, and re-add the task identifier of the download task that meets the retry conditions to the task queue. The scheduling service can take out the task identifier of the retried download task from the task queue, and re-call the corresponding download task to the download service and the parsing service based on the task identifier.

[0099] The above-mentioned method for processing download tasks receives download requests through access services, constructs a corresponding target download task for the download requests, and adds the task identifier of the target download task to the task queue. The access service makes the entire task processing engine highly scalable, and can efficiently access a large number of crawling requirements; after taking out the task identifier from the task queue through the scheduling service, the taken out task identifier is reported to the task status management service for task activation, and the taken out task identifier is scheduled to the download service and the parsing service; data is downloaded according to the target download task corresponding to the task identifier through the download service, and the task status is reported to the task status management service; the corresponding download data is obtained and parsed according to the task identifier through the parsing service, and the task status management service is reported to the task status management service. The service reports the task status, and based on this, the processing efficiency of the download task can be improved, and the robustness of the entire task processing engine can be ensured by dividing each function based on the data flow. The throughput of the entire task processing engine can be improved through task encapsulation, scheduling services, download services, and parsing services. In addition, through the task status management service, the download tasks that have been activated are monitored. When a download task whose corresponding task status meets the retry conditions is detected, the task identifier of the download task that meets the retry conditions is added back to the task queue. Based on this, the execution status of the download task can be grasped in real time through task status monitoring. The retry mechanism based on task status can ensure the integrity and success rate of the download task, making the task processing engine more reliable.

[0100] In one embodiment, the method also includes: obtaining a request rate range for an access site corresponding to a download request through a frequency control service; determining a task execution success rate for accessing the site based on a task status of a historical download task involving the access site reported by a download service and a resolution service through a frequency control service, and adjusting the request rate for accessing the site within the request rate range based on the task execution success rate; and scheduling the retrieved task identifier to the download service through a scheduling service, including: querying the request rate for accessing the site from the frequency control service through a scheduling service, and scheduling the retrieved task identifier to the download service according to the queried request rate.

[0101] Among them, the frequency control service is used to control the request rate of each access site. If the download request issued by the request object for a certain access site has a crawling demand of N times per second, or there are M download requests issued by different request objects for the same access site at the same time, when N or M is very large, if the download task is directly processed according to the frequency, it will be limited by the limited network bandwidth and the blocking policy of the access site, which will cause the download task execution of the same access site to fail. For this reason, in this embodiment, the frequency control service is used to adaptively control the request rate (rate) for each access site and adaptively control the download frequency for each access site, so that the entire task processing engine can automatically detect the characteristics of the access site, so as to achieve the optimal request rate at each access site under the network conditions and the restrictions of the access site.

[0102] The request rate refers to the rate at which network access requests are initiated to a certain access site, the success rate refers to the rate at which these network access requests successfully obtain return data, and the success rate is the ratio of the two, request rate × success rate = success rate. Fig.12 The figure shows the relationship between request rate, success rate, and success rate (i.e., task execution success rate). Fig.12 It can be seen that the first curve shows that the access success rate of the access site decreases with the increase of the request rate due to the network bandwidth and the blocking strategy of the access site. According to the relationship between the above three factors, the relationship between the success rate and the request rate (the second curve) can be obtained, that is, as the request rate increases, the success rate first increases and then decreases. In order to find a suitable set of success rates (Plow, Phigh) and a set of request rates (Vlow, Vhign), as Fig.12 As shown in the third curve of , by simply controlling the request rate between (Vlow, Vhigh), the success rate can fluctuate around the highest success rate Vsm.

[0103] The server can use the frequency control service to set the corresponding request rate interval according to the characteristics of each access site. The interval is determined by the minimum request rate (initial value) and the maximum request rate, denoted as (Vinit, Vmax]. The request rate for the access site will start from Vinit and be limited to the range of (Vinit, Vmax]. For example, authoritative websites and large websites have good performance in terms of throughput, so the corresponding request rate can be set relatively high. Small websites do not have good performance in terms of throughput, so the corresponding request rate can be set relatively low. Of course, the request rate interval of each access site can also be adjusted according to actual conditions. For example, if the current load of the access site is very high and the number of visits is large, the corresponding request rate interval can be lowered to avoid the access site from crashing due to inability to handle a large number of requests.

[0104] After the scheduling service takes out the task identifier from the task queue, it will query the request rate of the access site from the frequency control service according to the access site to be visited by the download task corresponding to the task identifier, and call the download service to process the download task according to the queried request rate. As mentioned above, the download service and the resolution service will report the task status to the task status management service. In this embodiment, the download service and the resolution service will also report the task status to the frequency control service. In this way, the frequency control service can count the task execution success rate of the download task related to the access site in the past period of time for each access site, and adjust the request rate of the access site within the above request rate range according to the statistical task execution success rate. For example, when the task execution success rate of the download task related to the access site in the past period of time decreases more, the request rate of the access site can be reduced. When the task execution success rate of the download task related to the access site in the past period of time increases more, the request rate of the access site can be appropriately increased. In this way, the entire link forms a feedback closed loop, thereby continuously adjusting the request rate of each access site, so that the task execution success rate is stabilized at a higher level.

[0105] like Fig.13 FIG. 1 is a functional diagram of a frequency control service in an embodiment. Fig.13 , each time the access service receives a download request for a new access site, it will initiate site registration with the frequency control service based on the access site, that is, the frequency control service will set a request rate range for the access site, and then the frequency control service will count the task execution success rate of the access site based on the task status reported by the download service and the resolution service, adjust the request rate of the corresponding access site based on the task execution success rate, and update it. The scheduling service can query the request rate of the access site from the frequency control service and schedule the download task for the access site according to the queried request rate.

[0106] In one embodiment, after the task identifier is retrieved from the task queue through the scheduling service, the method also includes: querying the download parameters of the target download task corresponding to the retrieved task identifier; if there is a historical download task whose corresponding download parameters are similar to the download parameters of the target download task, querying whether there is a cached parsing result corresponding to the historical download task; in the case where there is a cached parsing result corresponding to the historical download task, indexing and associating the result of the cached parsing result to the task identifier of the target download task, adding the result index to the parsing result queue, and reporting a message that the task parsing is successful corresponding to the task identifier to the task status management service; in the case where there is no cached parsing result corresponding to the historical task, executing the steps of reporting the retrieved task identifier to the task status management service for task activation, and scheduling the retrieved task identifier to the download service and the parsing service.

[0107] In this embodiment, in order to maximize the use of the parsing results, the server can also provide a cache service, and the cache service will record the parsing results of the download task, the result index of the parsing results, and the download parameters of the download task. After the scheduling service takes out the task identifier from the task queue, it can query the cache service whether there are historical download tasks with similar download parameters to the download parameters of the target download task according to the download parameters of the target download task corresponding to the task identifier. For example, a download task with the same access interface identifier and similar download keywords. If it exists, the corresponding parsing result may have been cached. If it has been cached, the cache service will feedback the corresponding result index to the scheduling service. The scheduling service will directly add the result index to the parsing result queue and report the task status management service that the task corresponding to the task identifier has been parsed successfully. In addition, in the case where there is a timeliness requirement for the target download task, the cache service also needs to check whether the timeliness of the cached parsing result meets the requirement. If there is no corresponding cached data or it does not meet the timeliness requirement, the scheduling service will directly dispatch the task identifier of the target download task to the download service and the parsing service for subsequent processing. Fig.14 FIG. 1 is a functional diagram of a cache service in an embodiment.

[0108] Fig.15 FIG. 1 is a flow chart of a method for processing a download task in an embodiment. Fig.15 , the processing method of the download task includes the following steps:

[0109] 1. Receive a download request through the access service. The download request carries download parameters. The download parameters include download keywords and the access interface identifier of the access site. Generate a corresponding task identifier for the download request through the access service, encapsulate the task according to the task identifier and download parameters, obtain the target download task, and add the task identifier of the target download task to the task queue;

[0110] 2. After taking out the task identifier from the task queue through the scheduling service, report the taken out task identifier to the task status management service for task activation, and schedule the taken out task identifier to the download service and the parsing service;

[0111] 3. Query the download parameters of the target download task corresponding to the retrieved task identifier; if there is a historical download task whose corresponding download parameters are similar to the download parameters of the target download task, query whether there is a cached parsing result corresponding to the historical download task;

[0112] 4. If there is a cached parsing result corresponding to the historical download task, the result index of the cached parsing result is associated with the task identifier of the target download task, and the result index is added to the parsing result queue;

[0113] 5. In the case that there is no cached parsing result corresponding to the historical task, data is downloaded through the download service according to the target download task corresponding to the task identifier;

[0114] 6. Generate a data index for the downloaded data and associate it with the task identifier, add the generated data index to the downloaded data queue, and report the task status to the task status management service;

[0115] 7. Obtain and parse the corresponding downloaded data according to the data index through the parsing service, obtain the parsing result, add it to the parsing result queue, and report the task status to the task status management service;

[0116] 8. Report the task status to the task status management service;

[0117] 9. The request object consumes the parsing results from the parsing result queue;

[0118] 10. The cache service accesses the parsing result queue and caches the parsing results.

[0119] 11. Monitor and store the task status of the download task through the task status management service.

[0120] The detailed description of each of the above steps has been presented in the previous text.

[0121] Fig.16 FIG. 1 is a schematic diagram of the system structure of a task processing engine in an embodiment. Fig.16 The system includes a task status management service, an access service, a scheduling service, a frequency control service, a download service, a resolution service, a cache service, and a dynamic proxy service. The task status management service is used to record the task status, query the task status of the download task, and perform a global task retry based on the task status. The access service is used to generate a download task, generate a task identifier for the download task, and deduplicate the download request. The scheduling service is used to manage the task queue, isolate the download tasks of different request objects, and prioritize the download tasks of the same request object. The scheduling service is also used to query the resolution results cached by the cache service. The scheduling service is also used to schedule tasks to the download service and the resolution service. The resolution service is used to parse the download data to obtain useful and structured resolution results, and is also used to initiate secondary download requests. The frequency control service is used to collect task status and adaptively adjust the request rate of each access site.

[0122] Fig.17 FIG. 1 is a schematic diagram of the scheduling relationship between various services in an embodiment. Fig.17The various services provided by the system are mainly centered on the dispatching service, and various services are called. Mainly, the dispatching service can call the cache service, task status management service, download service, and resolution service. The frequency control service feeds back the request rate of accessing the site to the dispatching service. The resolution service and download service report the task status to the frequency control service. The resolution service sends a secondary download request to the access service, and the download service calls the dynamic proxy service to download data.

[0123] Fig.18 This is a comparative diagram of the performance improvement of the task processing engine. Fig.18 Compared with the conventional scheme, the task failure rate and time consumption are significantly reduced. By monitoring the task status of the download task that has been activated, when the download task whose corresponding task status meets the retry conditions is detected, the task identifier of the download task that meets the retry conditions is added to the task queue again. Compared with no retry, the failure rate and time consumption of the task that is retried once are also significantly reduced.

[0124] The processing method of the download task provided in the embodiment of the present application splits each service according to the data flow, builds a sharp engine architecture, performs high-performance crawling of the system through scheduling services, download services, and parsing services, improves the overall efficiency and performance of the system crawling through adaptive frequency control services, generates web page snapshots through cache services to effectively perform efficient management, forms a multi-threaded, multi-domain distributed architecture for the task processing engine through task status management services, and makes the task management degree reach the limit, performs disaster recovery and storage design, and improves overall performance. Specifically, each service is split according to the data flow, builds a sharp engine architecture, enhances the robustness of the crawling engine, improves the throughput of the engine through task encapsulation, download scheduling, and parsing scheduling, achieves the optimal crawling speed through adaptive frequency control services, improves network bandwidth utilization through caching, achieves high reliability through disaster recovery design, and can also make the engine highly scalable through access service design.

[0125] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] Based on the same inventive concept, the embodiment of the present application also provides a download task processing device for implementing the download task processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more download task processing devices provided below can refer to the limitations of the download task processing method above, and will not be repeated here.

[0127] In one embodiment, Fig.19 As shown, a download task processing device 1900 is provided, comprising: a task access module 1902, a task scheduling module 1904, a task processing module 1906 and a status monitoring module 1908, wherein:

[0128] The task access module 1902 is used to receive a download request through an access service, construct a corresponding target download task for the download request, and add a task identifier of the target download task to a task queue;

[0129] The task scheduling module 1904 is used to retrieve the task identifier from the task queue through the scheduling service, report the retrieved task identifier to the task status management service for task activation, and schedule the retrieved task identifier to the download service and the parsing service;

[0130] The task processing module 1906 is used to download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service;

[0131] The status monitoring module 1908 is used to monitor the activated download tasks through the task status management service. When a download task whose corresponding task status meets the retry condition is monitored, the task identifier of the download task that meets the retry condition is added back to the task queue.

[0132] In one embodiment, the task access module 1902 is also used to receive a download request through an access service. The download request carries download parameters, and the download parameters include download keywords and an access interface identifier for accessing the site; a corresponding task identifier is generated for the download request through the access service, and the task is encapsulated according to the task identifier and the download parameters to obtain a structured task.

[0133] In one embodiment, the task access module 1902 is also used to generate a task identifier for a download request through an access service; when it is determined that the multiple download requests are for the same access site based on the respective download parameters in the multiple download requests, the same task group identifier is assigned to the multiple download requests; for each download request in the multiple download requests, the task is encapsulated based on the task group identifier, the task identifier and the download parameters to obtain multiple structured tasks belonging to the same task group.

[0134] In one embodiment, the task access module 1902 is also used to check the download keywords corresponding to multiple structured tasks belonging to the same task group; if there are at least two structured tasks with similar corresponding download keywords among the multiple structured tasks belonging to the same task group, deduplication processing is performed on the at least two structured tasks.

[0135] In one embodiment, the task access module 1902 is also used to register the task with the task status management service through the access service according to the task identifier of the target download task before adding the task identifier of the target download task to the task queue, so as to instruct the task status management service to persist the target download task.

[0136] In one embodiment, the download request carries a request object identifier and a request priority; the task access module 1902 is also used to determine multiple task queues corresponding to the request object identifier; determine a target queue that matches the request priority from the multiple task queues; and add the task identifier of the target download task to the target queue.

[0137] In one embodiment, the task processing module 1906 is also used to download data according to the target download task corresponding to the task identifier through the download service, add the data index of the downloaded data to the download data queue, and report the task status to the task status management service; retrieve the data index from the download data queue through the scheduling service, and schedule the retrieved data index to the parsing service; obtain and parse the corresponding download data according to the data index through the parsing service, add the result index of the parsing result to the parsing result queue, and report the task status to the task status management service.

[0138] In one embodiment, the task processing module 1906 is also used to download data through the download service according to the target download task corresponding to the task identifier; in the case of successful download, a data index is generated for the downloaded data and associated with the task identifier, the generated data index is added to the download data queue, and a message indicating that the task download corresponding to the task identifier is successfully downloaded is reported to the task status management service; in the case of failed download, a message indicating that the task download corresponding to the task identifier is failed is reported to the task status management service.

[0139] In one embodiment, the task processing module 1906 is also used to obtain and parse the corresponding downloaded data according to the data index through the parsing service; in the case of successful parsing, a result index is generated for the parsing result and associated with the task identifier, the generated result index is added to the parsing result queue, and a message indicating that the task parsing is successful corresponding to the task identifier is reported to the task status management service; in the case of failed parsing, a message indicating that the task parsing is unsuccessful corresponding to the task identifier is reported to the task status management service.

[0140] In one embodiment, the task processing module 1906 is also used to construct a download request according to the access interface identifier of the secondary access site when the parsing result includes the access interface identifier of the secondary access site belonging to the visited site, and send the constructed download request to the access service.

[0141] In one embodiment, the apparatus further comprises:

[0142] The frequency control module is used to obtain the request rate range of the access site corresponding to the download request through the frequency control service; determine the task execution success rate of the access site according to the task status of the historical download task involving the access site reported by the download service and the resolution service through the frequency control service, and adjust the request rate for the access site within the request rate range according to the task execution success rate;

[0143] The task scheduling module 1904 is further used to query the request rate for accessing the site from the frequency control service, and schedule the retrieved task identifier to the download service according to the queried request rate.

[0144] In one embodiment, the task scheduling module 1904 is also used to query the download parameters of the target download task corresponding to the retrieved task identifier; if there is a historical download task whose corresponding download parameters are similar to the download parameters of the target download task, it is queried whether there is a cached parsing result corresponding to the historical download task; in the case where there is a cached parsing result corresponding to the historical download task, the result index of the cached parsing result is indexed and associated with the task identifier of the target download task, the result index is added to the parsing result queue, and a message indicating that the task parsing is successful corresponding to the task identifier is reported to the task status management service; in the case where there is no cached parsing result corresponding to the historical task, the retrieved task identifier is reported to the task status management service for task activation, and the retrieved task identifier is scheduled to the download service and the parsing service.

[0145] The processing device of the above-mentioned download task receives the download request through the access service, constructs the corresponding target download task for the download request, and adds the task identifier of the target download task to the task queue. The access service makes the whole task processing engine highly scalable, and can efficiently access a large number of crawling requirements; after taking out the task identifier from the task queue through the scheduling service, it reports the taken out task identifier to the task status management service for task activation, and schedules the taken out task identifier to the download service and the parsing service; downloads data according to the target download task corresponding to the task identifier through the download service, reports the task status to the task status management service, obtains and parses the corresponding download data according to the task identifier through the parsing service, and reports to the task status management service. The service reports the task status, and based on this, the processing efficiency of the download task can be improved, and the robustness of the entire task processing engine can be ensured by dividing each function based on the data flow. The throughput of the entire task processing engine can be improved through task encapsulation, scheduling services, download services, and parsing services. In addition, through the task status management service, the download tasks that have been activated are monitored. When a download task whose corresponding task status meets the retry conditions is detected, the task identifier of the download task that meets the retry conditions is added back to the task queue. Based on this, the execution status of the download task can be grasped in real time through task status monitoring. The retry mechanism based on task status can ensure the integrity and success rate of the download task, making the task processing engine more reliable.

[0146] Each module in the processing device for the above-mentioned download task can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0147] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 20As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store task information related to the download task. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for processing a download task is implemented.

[0148] Those skilled in the art will understand that Fig. 20 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for processing a download task provided in an embodiment of the present application are implemented.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing a download task provided in an embodiment of the present application are implemented.

[0151] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method for processing a download task provided in an embodiment of the present application.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing a download task, characterized in that: The method comprises: Receive a download request through an access service, construct a corresponding target download task for the download request, and add a task identifier of the target download task to a task queue; After taking out the task identifier from the task queue through the scheduling service, the taken out task identifier is reported to the task status management service for task activation, and the taken out task identifier is scheduled to the download service and the parsing service; Download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service; The task status management service is used to monitor the activated download tasks. When a download task whose corresponding task status satisfies the retry condition is detected, the task identifier of the download task that satisfies the retry condition is added back to the task queue.

2. The method according to claim 1, characterized in that The receiving a download request through the access service and constructing a corresponding target download task for the download request includes: Receiving a download request through an access service, the download request carrying download parameters, the download parameters including a download keyword and an access interface identifier of an access site; A corresponding task identifier is generated for the download request through the access service, and the task is encapsulated according to the task identifier and the download parameters to obtain a structured task.

3. The method according to claim 2, characterized in that The step of generating a corresponding task identifier for the download request through the access service, and encapsulating the task according to the task identifier and the download parameter to obtain a structured task includes: generating a task identifier for the download request through an access service; In a case where it is determined according to the respective download parameters in the multiple download requests that the multiple download requests are for the same access site, assigning the same task group identifier to the multiple download requests; For each of the multiple download requests, task encapsulation is performed according to the task group identifier, the task identifier and the download parameter to obtain multiple structured tasks belonging to the same task group.

4. The method according to claim 3, characterized in that The method further comprises: Check the download keywords corresponding to multiple structured tasks belonging to the same task group; If there are at least two structured tasks with similar corresponding download keywords among the multiple structured tasks belonging to the same task group, deduplication processing is performed on the at least two structured tasks.

5. The method according to claim 1, characterized in that Before adding the task identifier of the target download task to the task queue, the method further includes: The task is registered with the task status management service according to the task identifier of the target download task through the access service to instruct the task status management service to persist the target download task. After receiving the notification of successful persistence, the step of adding the task identifier of the target download task to the task queue is executed.

6. The method according to claim 1, characterized in that The download request carries a request object identifier and a request priority; The step of adding the task identifier of the target download task to the task queue includes: Determine a plurality of task queues corresponding to the request object identifier; Determine a target queue matching the request priority from the multiple task queues; Add the task identifier of the target download task to the target queue.

7. The method according to claim 1, characterized in that The downloading service downloads data according to the target download task corresponding to the task identifier, reports the task status to the task status management service, obtains and parses the corresponding download data according to the task identifier through the parsing service, and reports the task status to the task status management service, including: Download data according to the target download task corresponding to the task identifier through the download service, add the data index of the downloaded data to the download data queue, and report the task status to the task status management service; fetching a data index from the download data queue through the scheduling service, and dispatching the fetched data index to the parsing service; The parsing service obtains and parses the corresponding download data according to the data index, adds the result index of the parsing result to the parsing result queue, and reports the task status to the task status management service.

8. The method according to claim 7, characterized in that The downloading service downloads data according to the target download task corresponding to the task identifier, adds the data index of the downloaded data to the download data queue, and reports the task status to the task status management service, including: Downloading data through the download service according to the target download task corresponding to the task identifier; In the case of successful download, a data index is generated for the downloaded data and associated with the task identifier, the generated data index is added to the download data queue, and a message that the task corresponding to the task identifier has been successfully downloaded is reported to the task status management service; The method further comprises: In the case of download failure, a message indicating that the task download corresponding to the task identifier has failed is reported to the task status management service.

9. The method according to claim 7, characterized in that: The obtaining and parsing corresponding download data according to the data index through the parsing service, adding the result index of the parsing result to the parsing result queue, and reporting the task status to the task status management service include: Acquire and parse corresponding download data according to the data index through the parsing service; If the parsing is successful, a result index is generated for the parsing result and associated with the task identifier, the generated result index is added to the parsing result queue, and a message that the task parsing corresponding to the task identifier is successful is reported to the task status management service; The method further comprises: In the case of parsing failure, a message indicating that the task parsing corresponding to the task identifier has failed is reported to the task status management service.

10. The method according to claim 7, characterized in that After acquiring and parsing the corresponding downloaded data according to the data index through the parsing service and adding the result index of the parsing result to the parsing result queue, the method further includes: When the parsing result includes the access interface identifier of the secondary access site belonging to the visited site, a download request is constructed according to the access interface identifier of the secondary access site, and the constructed download request is sent to the access service.

11. The method according to claim 1, characterized in that: The method further comprises: Obtaining, through a frequency control service, a request rate interval for an access site corresponding to the download request; Determine, by the frequency control service, a task execution success rate for the access site according to the task status of the historical download tasks involving the access site reported by the download service and the resolution service, and adjust the request rate for the access site within the request rate range according to the task execution success rate; The step of dispatching the retrieved task identifier to the download service includes: The request rate for the access site is queried from the frequency control service, and the retrieved task identifier is dispatched to the download service according to the queried request rate.

12. The method according to any one of claims 1 to 11, characterized in that: After taking out the task identifier from the task queue through the scheduling service, the method further includes: Query the download parameters of the target download task corresponding to the retrieved task identifier; If there is a historical download task whose corresponding download parameters are similar to the download parameters of the target download task, query whether there is a cached parsing result corresponding to the historical download task; In the case where there is a cached parsing result corresponding to the historical download task, the result index of the cached parsing result is associated with the task identifier of the target download task, the result index is added to the parsing result queue, and a message that the task parsing corresponding to the task identifier is successful is reported to the task status management service; In the case that there is no cached parsing result corresponding to the historical task, the steps of reporting the retrieved task identifier to the task status management service to activate the task and dispatching the retrieved task identifier to the download service and the parsing service are performed.

13. A download task processing device, characterized in that: The device comprises: A task access module, used to receive a download request through an access service, construct a corresponding target download task for the download request, and add a task identifier of the target download task to a task queue; A task scheduling module, used for taking out a task identifier from the task queue through a scheduling service, reporting the taken out task identifier to a task status management service for task activation, and scheduling the taken out task identifier to a download service and a parsing service; The task processing module is used to download data according to the target download task corresponding to the task identifier through the download service, report the task status to the task status management service, obtain and parse the corresponding download data according to the task identifier through the parsing service, and report the task status to the task status management service; The status monitoring module is used to monitor the activated download tasks through the task status management service. When a download task whose corresponding task status meets the retry condition is monitored, the task identifier of the download task that meets the retry condition is added back to the task queue.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images