Distributed crawler method based on service grid, medium and equipment

Abstract

The invention discloses a distributed crawler method based on a service grid, a medium and equipment, and the method comprises the steps: triggering a fusing mechanism if the proportion of failure requests in unit time exceeds a preset proportion, and automatically fusing all crawler requests for a target website; and after the fusing recovery condition is met, re-receiving the first number of crawler requests, counting a first response success rate of the target website to the first number of crawler requests, and if the first response success rate is greater than a preset first success rate, re-receiving a second number of crawler requests, the second response success rate of the target website to a second number of crawler requests is counted, if the second response success rate is larger than a preset second success rate, the fusing mechanism is stopped, responses of the target website to all the crawler requests are recovered, and the second number is larger than the first number. By setting the fusing mechanism, paralysis of the whole crawler system caused by the problem of a certain website can be effectively prevented, and the stability of the crawler is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a distributed crawler method, medium and device based on a service grid. Background Art

[0002] Traditional crawler technology solutions mainly rely on developers to write code to implement network requests, data parsing, task scheduling and other functions. Developers need to use the network library provided by the programming language (such as Python's Requests library) to build HTTP requests and implement task queues, concurrency control and other mechanisms to improve crawling efficiency and process server responses. This traditional crawler technology solution has some inherent problems, one of which is the connection timeout problem.

[0003] Due to reasons such as excessive server load and network bandwidth limitations, the target website may respond slowly, causing the crawler request to time out. Traditional crawler codes usually lack a complete error handling and retry mechanism, and once a network problem occurs, it is easy to cause program interruption or timeout. Summary of the invention

[0004] In view of the above problems, the present application provides a technical solution for a distributed crawler based on a service grid to solve the problems of request timeout, low crawler efficiency and stability existing in traditional crawler technology.

[0005] To achieve the above objectives, in a first aspect, the present application provides a distributed crawler method based on a service grid, the method comprising:

[0006] Configure the preset time in the sidecar proxy of the service mesh;

[0007] Monitor all crawler requests to the target website, and if the response time of the target website to a crawler request exceeds the preset time, mark the crawler request as a failed request;

[0008] If the proportion of failed requests per unit time exceeds a preset ratio, the circuit breaker mechanism is triggered to automatically circuit breaker all crawler requests to the target website; and

[0009] After the circuit breaker recovery conditions are met, the first number of crawler requests are received again, and the first response success rate of the target website to the first number of crawler requests is counted; if the first response success rate is greater than the preset first success rate, the second number of crawler requests are received again, and the second response success rate of the target website to the second number of crawler requests is counted; if the second response success rate is greater than the preset second success rate, the circuit breaker mechanism is stopped, and the target website's response to all crawler requests is restored, and the second number is greater than the first number.

[0010] Further, the method further includes: configuring a preset number of retries in a Sidecar proxy of the service grid;

[0011] If the target website's response time to a crawler request exceeds the preset time, the method further includes:

[0012] Re-initiate a crawler request to connect to the target website, and if the number of connection failures reaches the preset number of retries, mark the crawler request as a failed request.

[0013] Furthermore, the method further comprises:

[0014] If the number of failed requests monitored within a unit time exceeds a first preset number, the concurrent request volume for the current target website is dynamically reduced.

[0015] Furthermore, the method further comprises:

[0016] Establish a failed request list, and add the crawler request marked as a failed request to the failed request list;

[0017] When it is detected that the number of all crawler requests to the target website is less than a second preset number, the crawler requests in the failed request list are obtained and re-executed.

[0018] Furthermore, the unit time is a preset time period closest to the current timestamp, and is dynamically updated as the current timestamp changes.

[0019] Furthermore, the first quantity and the second quantity are determined according to the following method:

[0020] Input the current timestamp, the number of all crawler requests, and the number of failed requests into the trained first neural network model, and output a first number;

[0021] The first quantity, the current timestamp, the number of all crawler requests, and the number of failed requests are input into a trained second neural network model, and a second quantity is output.

[0022] Furthermore, the first neural network model that has been trained is trained according to the following method:

[0023] Obtaining first sample information, and determining a first sample quantity according to the first sample information, wherein the first sample information includes a sample timestamp, a total number of sample crawler requests, and a number of sample failed requests;

[0024] Initiate a crawler request for the first sample quantity to the target website, count the response results of the target website to the first sample quantity, adjust the first sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal first sample quantity corresponding to the current first sample information is obtained, and the trained first neural network model is obtained.

[0025] Furthermore, the trained second neural network model is trained according to the following method:

[0026] Acquire second sample information, and determine a second sample quantity according to the second sample information, wherein the second sample information includes the first sample quantity and the first sample information;

[0027] Initiate a crawler request for the second sample quantity to the target website, count the response results of the target website to the second sample quantity, adjust the second sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal second sample quantity corresponding to the current second sample information is obtained, and the trained second neural network model is obtained.

[0028] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed crawler method based on a service grid as described in the first aspect of the present application.

[0029] In a third aspect, the present application provides an electronic device having a computer program stored thereon, including a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the distributed crawler method based on the service grid as described in the first aspect of the present application is implemented.

[0030] Different from the prior art, the above technical solution involves a distributed crawler method, medium and device based on a service grid, which includes: configuring a preset time in the Sidecar proxy of the service grid; monitoring all crawler requests to the target website, and if the response time of the target website to a crawler request exceeds the preset time, marking the crawler request as a failed request; if the proportion of failed requests per unit time exceeds the preset proportion, triggering the fuse mechanism, automatically fusing all crawler requests to the target website; and after meeting the fuse recovery conditions, re-receiving the first number of crawler requests, counting the first response success rate of the target website to the first number of crawler requests, if the first response success rate is greater than the preset first success rate, re-receiving the second number of crawler requests, counting the second response success rate of the target website to the second number of crawler requests, if the second response success rate is greater than the preset second success rate, stopping the fuse mechanism, restoring the target website's response to all crawler requests, and the second number is greater than the first number. By setting a fuse mechanism, the entire crawler system can be effectively prevented from being paralyzed due to problems on a certain website, thereby improving the stability of the crawler.

[0031] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0033] In the drawings of the specification:

[0034] Figure 1 This is a flow chart of a distributed crawler method based on a service grid as described in the first exemplary embodiment of the present application;

[0035] Figure 2 This is a flow chart of a distributed crawler method based on a service grid according to a second exemplary embodiment of the present application;

[0036] Figure 3 This is a flow chart of a distributed crawler method based on a service grid according to a third exemplary embodiment of the present application;

[0037] Figure 4 This is a flow chart of a distributed crawler method based on a service grid according to a fourth exemplary embodiment of the present application;

[0038] Figure 5This is a flow chart of a distributed crawler method based on a service grid according to a fifth exemplary embodiment of the present application;

[0039] Figure 6 This is a flow chart of a distributed crawler method based on a service grid according to a sixth exemplary embodiment of the present application;

[0040] Figure 7 A schematic diagram of an electronic device according to an exemplary embodiment of the present application;

[0041] The reference numerals in the above drawings are described as follows:

[0042] 10. Electronic equipment;

[0043] 101. Processor;

[0044] 102. Storage medium. DETAILED DESCRIPTION

[0045] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0046] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0047] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0048] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0049] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0050] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0051] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0052] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0053] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0054] In the first aspect, Figure 1 As shown, the present application provides a distributed crawler method based on a service grid, the method comprising:

[0055] First, go to step S101 to configure the preset time in the Sidecar proxy of the service grid;

[0056] Then proceed to step S102 to monitor all crawler requests to the target website. If the response time of the target website to a crawler request exceeds the preset time, the crawler request is marked as a failed request;

[0057] Then, step S103 is entered. If the proportion of failed requests per unit time exceeds a preset proportion, a fuse mechanism is triggered to automatically fuse all crawler requests to the target website; and

[0058] Then enter step S104. After the fuse recovery conditions are met, the first number of crawler requests are received again, and the first response success rate of the target website to the first number of crawler requests is counted. If the first response success rate is greater than the preset first success rate, the second number of crawler requests are received again, and the second response success rate of the target website to the second number of crawler requests is counted. If the second response success rate is greater than the preset second success rate, the fuse mechanism is stopped, and the target website's response to all crawler requests is restored. The second number is greater than the first number.

[0059] In step S101, Service Mesh is an architectural pattern for managing and controlling the communication between services in a microservice architecture. It provides infrastructure functions such as load balancing, service discovery, traffic management, security, etc. by adding a dedicated layer (usually called a data plane) between service instances. The core goal of the service mesh is to simplify the complexity in the microservice architecture and improve the reliability, elasticity and observability of the system. The components of the service mesh include the data plane and the control plane. The data plane is responsible for actually handling the communication between services. It is usually composed of proxies (such as Envoy, Linkerd, etc.), which run next to each service instance, intercept all inbound and outbound traffic, and perform various network functions.

[0060] The control plane is responsible for managing and configuring the data plane. It provides functions such as service discovery, configuration management, policy management, and telemetry collection.

[0061] Sidecar proxy refers to a design pattern that is commonly used in microservice architectures and service meshes. It runs next to each service instance as a separate process or container, providing management and control of the network traffic for that service. Sidecar proxy is usually responsible for handling various functions related to inter-service communication, such as load balancing, service discovery, security, monitoring, and traffic management.

[0062] The preset time refers to the request timeout. By setting the request timeout, you can determine whether the response time of each crawler request has timed out.

[0063] In step S104, by configuring the fuse rule, when the connection timeout ratio exceeds a certain threshold, the request to the target website is automatically fused. After a period of fusion, you can try to allow a small number of requests to pass through for trial. If successful, continue to increase the number of trial requests. If it still passes, it will return to normal and stop the fuse mechanism. Otherwise, continue to fuse. The fuse mechanism can effectively prevent the entire crawler system from being paralyzed due to a problem with a certain website, and also leave time for the target website to recover.

[0064] In some embodiments, Figure 2 As shown, the method also includes:

[0065] Step S201: Configure a preset number of retries in the Sidecar proxy of the service grid;

[0066] Step S202: if the response time of the target website to a crawler request exceeds the preset time, re-initiate the crawler request to connect to the target website, and if the number of connection failures reaches the preset number of retries, mark the crawler request as a failed request.

[0067] In short, by setting a reasonable timeout period, you can avoid blocking the entire crawler process due to long-term unresponsiveness of individual requests. At the same time, setting the number of retries allows Sidecar to make multiple attempts when the first request fails, thereby increasing the probability of successful connection.

[0068] In some embodiments, the method further includes: if the number of failed requests monitored within a unit time exceeds a first preset number, dynamically reducing the concurrent request volume for the current target website.

[0069] Specifically, you can monitor the indicator data collected by the Sidecar proxy of the service grid, such as request latency and success rate. When an increase in connection timeout problems is detected, the request concurrency can be dynamically reduced to avoid excessive pressure on the target website and reduce the waste of resources caused by the accumulation of a large number of timed out requests. The Sidecar proxy of the service grid is responsible for collecting request-related indicator data and reporting it to the monitoring system. The control plane can dynamically adjust the configuration of the Sidecar based on the monitoring data to control the request concurrency.

[0070] In some embodiments, Figure 3 As shown, the method also includes:

[0071] Step S301: Create a failed request list, and add the crawler request marked as a failed request to the failed request list;

[0072] Step S302: when it is detected that the number of all crawler requests to the target website is less than a second preset number, the crawler requests in the failed request list are obtained and re-executed.

[0073] In short, by first separating the failed requests, the impact on the target website's crawler is avoided, and the normal progress of other requests is ensured, and these requests can be executed after the pressure on the target website is reduced. Of course, in other embodiments, the timestamp information of each failed request can also be recorded, and if the timestamp information is far from the current timestamp, the processing can be directly abandoned.

[0074] In some embodiments, the unit time is a preset time period closest to the current timestamp, and is dynamically updated as the current timestamp changes.

[0075] In short, when counting whether the crawler request is successful, it is preferred to count the preset time period closest to the current timestamp. This can more truly reflect the response status of the current target website to each crawler request, and then determine whether a fuse mechanism is needed for processing based on the response status.

[0076] like Figure 4 As shown, in some embodiments, the first quantity and the second quantity are determined according to the following method:

[0077] Step S401: input the current timestamp, the number of all crawler requests, and the number of failed requests into the trained first neural network model, and output a first number;

[0078] Step S402: input the first number, the current timestamp, the number of all crawler requests and the number of failed requests into the trained second neural network model, and output the second number.

[0079] Further, such as Figure 5 As shown, the first neural network model that has been trained is trained according to the following method:

[0080] Step S501: obtaining first sample information, and determining a first sample quantity according to the first sample information, wherein the first sample information includes a sample timestamp, a total number of sample crawler requests, and a number of sample failed requests;

[0081] Step S502: initiate a crawler request for the first sample quantity to the target website, count the response results of the target website to the first sample quantity, adjust the first sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal first sample quantity corresponding to the current first sample information is obtained, and the trained first neural network model is obtained.

[0082] Further, such as Figure 6 As shown, the trained second neural network model is trained according to the following method:

[0083] S601: Acquire second sample information, and determine a second sample quantity according to the second sample information, where the second sample information includes the first sample quantity and the first sample information;

[0084] S602: Initiate a crawler request for a second sample quantity to the target website, count the response results of the target website to the second sample quantity, adjust the second sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal second sample quantity corresponding to the current second sample information is obtained, and the trained second neural network model is obtained.

[0085] In this embodiment, the first neural network model can be a convolutional neural network (CNN) or a recurrent neural network (RNN) and its variants (such as LSTM, GRU).

[0086] In this embodiment, the response result of the target website to the first sample quantity refers to the success rate ratio of the target website's response to the first sample quantity. If the success rate is higher than the set value, the first sample quantity can be further increased to make the set fuse recovery condition more accurate. Otherwise, the first sample quantity can be reduced. The optimal first sample quantity is a comprehensive result that fully considers the total number of crawler requests for historical visits in each time period and the success rate of the response. What it wants to achieve is that on the basis of meeting a certain response success rate, the total number of crawler requests is as large as possible, which can be calculated by setting a weight score. The second sample quantity is the same, and will not be elaborated here.

[0087] In short, the specific values ​​of the first number and the second number are not fixed, but are the results obtained by training the neural network model based on a large amount of data. For example, if it is found that there are usually a large number of crawler requests within a certain period of time, then when setting the conditions for circuit breaker recovery, the first number and second number that need to be tested will be adaptively improved, thereby making the judgment of whether the system has returned to normal more accurate.

[0088] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the distributed crawler method based on the service grid as described in the first aspect of the present invention is implemented.

[0089] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0090] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory can be a disk memory or a tape memory.

[0091] The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.

[0092] like Figure 7 As shown, in the third aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, on which a computer program is stored, and when the computer program is executed by the processor, the distributed crawler method based on the service grid as described in the first aspect of the present invention is implemented.

[0093] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use circuits, single or multiple application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), digital signal processing devices (Digital Signal Processing Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field Programmable Gate Array, FPGA), central processing units (Central Processing Unit, CPU), controllers, microcontrollers, at least one of microprocessors, so that the processor can execute some or all of the steps or any combination of the steps in the service grid-based distributed crawler method described in the various embodiments of the present application.

[0094] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A distributed crawler method based on a service grid, characterized in that: The method comprises: Configure the preset time in the sidecar proxy of the service mesh; Monitor all crawler requests to the target website, and if the response time of the target website to a crawler request exceeds the preset time, mark the crawler request as a failed request; If the proportion of failed requests per unit time exceeds a preset ratio, the circuit breaker mechanism is triggered to automatically circuit breaker all crawler requests to the target website; and After the circuit breaker recovery conditions are met, the first number of crawler requests are received again, and the first response success rate of the target website to the first number of crawler requests is counted; if the first response success rate is greater than the preset first success rate, the second number of crawler requests are received again, and the second response success rate of the target website to the second number of crawler requests is counted; if the second response success rate is greater than the preset second success rate, the circuit breaker mechanism is stopped, and the target website's response to all crawler requests is restored, and the second number is greater than the first number.

2. The distributed crawler method based on service grid as claimed in claim 1, characterized in that: The method further includes: configuring a preset number of retries in a Sidecar proxy of the service grid; If the target website's response time to a crawler request exceeds the preset time, the method further includes: Re-initiate a crawler request to connect to the target website, and if the number of connection failures reaches the preset number of retries, mark the crawler request as a failed request.

3. The distributed crawler method based on service grid as claimed in claim 1, characterized in that: The method further comprises: If the number of failed requests monitored within a unit time exceeds a first preset number, the concurrent request volume for the current target website is dynamically reduced.

4. The distributed crawler method based on service grid as claimed in claim 1, characterized in that: The method further comprises: Establish a failed request list, and add the crawler request marked as a failed request to the failed request list; When it is detected that the number of all crawler requests to the target website is less than a second preset number, the crawler requests in the failed request list are obtained and re-executed.

5. The distributed crawler method based on service grid as claimed in claim 1, characterized in that: The unit time is a preset time period closest to the current timestamp, and is dynamically updated as the current timestamp changes.

6. The distributed crawler method based on service grid as claimed in claim 1, characterized in that: The first quantity and the second quantity are determined according to the following method: Input the current timestamp, the number of all crawler requests, and the number of failed requests into the trained first neural network model, and output a first number; The first quantity, the current timestamp, the number of all crawler requests, and the number of failed requests are input into a trained second neural network model, and a second quantity is output.

7. The distributed crawler method based on service grid as claimed in claim 6, characterized in that: The first neural network model that has been trained is trained according to the following method: Obtaining first sample information, and determining a first sample quantity according to the first sample information, wherein the first sample information includes a sample timestamp, a total number of sample crawler requests, and a number of sample failed requests; Initiate a crawler request for the first sample quantity to the target website, count the response results of the target website to the first sample quantity, adjust the first sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal first sample quantity corresponding to the current first sample information is obtained, and the trained first neural network model is obtained.

8. The distributed crawler method based on service grid as claimed in claim 7, characterized in that: The trained second neural network model is trained according to the following method: Acquire second sample information, and determine a second sample quantity according to the second sample information, wherein the second sample information includes the first sample quantity and the first sample information; Initiate a crawler request for the second sample quantity to the target website, count the response results of the target website to the second sample quantity, adjust the second sample quantity according to the response results and re-initiate the crawler request, repeat the above steps until the optimal second sample quantity corresponding to the current second sample information is obtained, and the trained second neural network model is obtained.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed crawler method based on the service grid is implemented as described in any one of claims 1 to 8.

10. An electronic device having a computer program stored thereon, characterized in that: It comprises a processor and a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, a distributed crawler method based on a service grid is implemented as described in any one of claims 1 to 8.

Images