Network content crawling method, electronic device, and storage medium

Abstract

The application provides a network content crawling method, an electronic device and a storage medium. The network crawler comprises a main crawler, a task distribution platform and a plurality of sub-crawlers for processing different task types. The method comprises: the main crawler sending a target task type of a network content crawling task to the task distribution platform, wherein the target task type is determined according to task information carried in a task processing request of the network content crawling task; and the task distribution platform distributing the network content crawling task to a corresponding sub-crawler for processing according to the target task type. Since the network content crawling task is divided into task types and processed by corresponding sub-crawlers in the network content crawling process, the possibility of blocking can be reduced in the large-scale network content crawling process.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and more specifically, to web content crawling methods, electronic devices, and storage media. Background Technology

[0002] With the continuous development of science and technology, web crawlers are often used to scrape content from the internet. However, current web crawlers are prone to blocking during large-scale web content scraping. Summary of the Invention

[0003] The purpose of this application is to provide a web content crawling method, electronic device, and storage medium to solve the problems in the prior art.

[0004] The first aspect of this application provides a method for web content crawling, characterized in that the web crawler includes a main crawler, a task distribution platform, and multiple sub-crawlers for handling different task types, including:

[0005] The main crawler sends the target task type of the web content crawling task to the task distribution platform, wherein the target task type is determined according to the task information carried in the task processing request of the web content crawling task.

[0006] The task distribution platform assigns the web content crawling task to the corresponding sub-crawler for processing based on the target task type.

[0007] In one embodiment, the web crawler further includes a resource scheduling platform; and the method further includes:

[0008] The resource scheduling platform obtains the usage information of different types of resources in each sub-crawler;

[0009] The resource scheduling platform determines the task type to be processed by each sub-crawler based on the usage information of different types of resources in each sub-crawler.

[0010] In one embodiment, the types of resources include CPU resources, memory resources, disk resources, network bandwidth resources, and physical address resources; and,

[0011] Task types include CPU resource intensive tasks, memory resource intensive tasks, disk resource intensive tasks, network bandwidth resource intensive tasks, and physical address resource intensive tasks.

[0012] In one embodiment, the resource scheduling platform obtains the usage information of different types of resources in each sub-crawler, specifically including:

[0013] The resource scheduling platform obtains the usage information of different types of resources periodically reported by each sub-crawler.

[0014] In one embodiment, the method further includes:

[0015] The main crawler obtains the task processing request for the web content crawling task, and the task processing request carries the task information.

[0016] In one embodiment, the task processing request is entered by the user on a page displayed on the terminal; or,

[0017] Task processing requests are generated by triggering scheduled tasks.

[0018] In one embodiment, the main crawler sends the target task type of the web content crawling task to the task distribution platform, specifically including:

[0019] The main crawler sends the target task type of the web content crawling task to the task distribution platform through the message queue bus.

[0020] In one embodiment, the main crawler specifically includes: a main crawler based on the Node.js architecture.

[0021] A second aspect of this application provides a risk entity identification device for insurance claims, comprising:

[0022] A second aspect of this application provides an electronic device, including:

[0023] Memory, used to store computer programs;

[0024] A processor for executing the method described in any one of the embodiments of the present application.

[0025] A third aspect of this application provides a storage medium comprising: a program that, when run on an electronic device, enables the electronic device to execute any method described in the method embodiments of this application.

[0026] The web content crawling method provided in this application includes a main crawler, a task distribution platform, and multiple sub-crawlers for handling different task types. The method involves the main crawler sending the target task type of the web content crawling task to the task distribution platform. The target task type is determined based on the task information carried in the task processing request of the web content crawling task. Then, the task distribution platform allocates the web content crawling task to the corresponding sub-crawler for processing according to the target task type. Because the web content crawling task is categorized by task type and processed using corresponding sub-crawlers during the crawling process, the possibility of blocking can be reduced during large-scale web content crawling. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the specific structure of an electronic device provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of the specific structure of a web crawler provided in one embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating the interaction between a web crawler and a terminal, provided as an embodiment of this application.

[0031] Figure 4 A schematic diagram illustrating the specific process of a web content crawling method provided in an embodiment of this application;

[0032] Figure 5 A schematic diagram of the specific structure of another web crawler provided in one embodiment of this application;

[0033] Figure 6 A schematic diagram of the specific structure of another web crawler provided in one embodiment of this application;

[0034] Figure 7 This is a schematic diagram of the specific structure of a web content crawling device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. In the description of this application, terms such as "first," "second," and "third" are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0036] As mentioned earlier, with the continuous development of science and technology, it is often necessary to use web crawlers to scrape content from the internet. However, current web crawlers are prone to blocking during large-scale web content scraping.

[0037] Based on this, embodiments of this application provide a web content crawling method, electronic device, and storage medium, which can reduce the possibility of blocking during the web content crawling process. Figure 1The image shows an electronic device 1 provided in this embodiment. The electronic device 1 includes at least one processor 11 and a memory 12. Figure 1 Taking a processor as an example, the processor 11 and the memory 12 can be connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the process of the method in the following embodiments.

[0038] In practical applications, the electronic device 1 can be a mobile phone, laptop, desktop computer, or a large server or server cluster composed of them. For example, in the application scenario of this application embodiment, when crawling network content, the electronic device 1 can act as a server or server cluster.

[0039] In the embodiments of this application, such as Figure 2 As shown, the web crawler 2 of this application is deployed in a distributed structure. The web crawler 2 includes a main crawler 21, a task distribution platform 22, and multiple sub-crawlers 23. These sub-crawlers 23 can handle web content crawling tasks of different task types. In this web crawler 2, the main crawler 21 is communicatively connected to the task distribution platform 22, enabling data communication between the main crawler 21 and the task distribution platform 22. The task distribution platform 22 is also communicatively connected to each of the sub-crawlers 23, enabling data communication between the task distribution platform 22 and each of the sub-crawlers 23. Of course, the sub-crawlers 23 can also be deployed as a distributed cluster, thereby improving efficiency through the distributed cluster of sub-crawlers.

[0040] It should be noted that web content crawling tasks can be categorized into CPU-intensive, memory-intensive, disk-intensive, network bandwidth-intensive, and physical address-intensive tasks. Specifically, CPU-intensive tasks require significant CPU resources; memory-intensive tasks require significant memory resources; disk-intensive tasks require significant disk resources; network bandwidth-intensive tasks require significant network bandwidth resources; and physical address-intensive tasks require significant physical address resources.

[0041] In addition, such as Figure 3As shown, the main crawler 21 in web crawler 2 can also connect to one or more terminals 3. Terminal 3 can be a user's mobile phone or computer, or other server nodes. The main crawler 21 can communicate with each terminal 3. Therefore, the user can initiate a task processing request for web content crawling to the main crawler 21 through the terminal 3, and the main crawler 21 can receive the task processing request.

[0042] In the web crawler 2 provided in this embodiment, the main crawler 21 can be a main crawler developed based on the Node.js architecture. Because Node.js and JavaScript share the same origin, this main crawler can easily parse HTML-type task processing requests. Furthermore, during large-scale web content crawling, the natural advantages of Golang coroutines in multi-threaded processing can be utilized to significantly accelerate web content crawling efficiency with higher concurrency performance and lower resource consumption (Java's single-threaded overhead is 4MB, while Golang only requires 4KB). Node.js, released in May 2009 and developed by Ryan Dahl, is a JavaScript runtime environment based on the Chrome V8 engine, using an event-driven, non-blocking I / O model.

[0043] This application provides a method for web content crawling, which can be used to crawl web content. The method can be implemented by... Figure 1 The electronic device 1 shown is used to perform this action. For example... Figure 4 The diagram shows the specific flow of this method, which includes the following steps:

[0044] Step S41: The main crawler obtains the task processing request for the web content crawling task.

[0045] The task processing request carries task information, which allows the determination of the task type (referred to as the target task type) for the web content crawling task. This task information may include the web address (e.g., URL) of the web content to be crawled, the network channel, and a task type classification marker. For example, if the task information is the URL of the web content to be crawled, and the URL points to a large web platform, it indicates that a large amount of text, images, audio, and video content needs to be crawled. In this case, the web content crawling task will consume a lot of CPU resources, making it a CPU-intensive task. The task information can also be a task type classification marker, which can directly determine the corresponding task type. Furthermore, the task information can also be a network channel, such as a high-performance cluster channel, a medium-performance cluster channel, a low-performance cluster channel, a high-latency cluster channel, a medium-latency cluster channel, a low-latency cluster channel, a Shanghai dedicated channel, or a Beijing dedicated channel, thus determining the task type of the web content crawling task based on the network channel.

[0046] Additionally, in step S41, the user can typically input a task processing request for web content crawling from the terminal, thereby requesting that the web content be crawled. For example, the user can input the task processing request through touch, mouse clicks, or other means on the page displayed on the terminal; of course, the user can also set up a scheduled task, which will then send the task processing request to the main crawler when triggered by the scheduled task.

[0047] Accordingly, the main crawler can obtain task processing requests in several ways. First, it can directly receive task processing requests entered by the user through the page displayed on the triggering terminal. Second, it can obtain the request from a task processing request queue. For example, in practical applications, since the main crawler connects to multiple terminals, when multiple different users enter task processing requests through these terminals in a short period, the main crawler may receive a large number of task processing requests, leading to significant processing pressure. To reduce this pressure, a task processing request queue can be pre-built, and these requests can be added sequentially according to their sending time or arrival time at the main crawler. This allows the main crawler to obtain the request from the queue, thereby reducing its processing pressure.

[0048] Step S42: The main crawler determines the target task type of the web content crawling task based on the task information carried in the task processing request.

[0049] As mentioned above, the task processing request carries task information, which can be used to determine the target task type of the web content crawling task. Therefore, in step S42, the main crawler determines the target task type of the web content crawling task based on the task information carried in the task processing request. For example, the target task type can be a CPU resource intensive task, a memory resource intensive task, a network bandwidth resource intensive task, a physical address resource intensive task, or other types of tasks.

[0050] Step S43: The main crawler sends the target task type of the web content crawling task to the task distribution platform.

[0051] After determining the target task type for web content crawling, the main crawler can send the target task type to the task distribution platform. Of course, to facilitate subsequent web content crawling, it can also send information such as the network address of the web content to be crawled to the task distribution platform.

[0052] Step S44: The task distribution platform assigns web content crawling tasks to the corresponding sub-crawlers for processing based on the target task type.

[0053] The web crawler in this embodiment includes multiple sub-crawlers. Since these sub-crawlers can be used to handle web content crawling tasks of different task types, after receiving the target task type sent by the main crawler, the task distribution platform can determine the corresponding sub-crawler from among the sub-crawlers according to the target task type. The task type to be handled by the determined sub-crawler is the target task type. For example, if the target task type is a CPU resource-intensive task, then the task type to be handled by the sub-crawler is a CPU resource-intensive task; or if the target task type is a memory resource-intensive task, then the task type to be handled by the sub-crawler is a memory resource-intensive task. After determining the sub-crawler, the web content crawling task is assigned to the sub-crawler for processing.

[0054] The web content crawling method provided in this application includes a main crawler, a task distribution platform, and multiple sub-crawlers for handling different task types. The method includes the main crawler sending the target task type of the web content crawling task to the task distribution platform. The target task type is determined based on the task information carried in the task processing request of the web content crawling task. Then, the task distribution platform allocates the web content crawling task to the corresponding sub-crawler for processing according to the target task type. Since the web content crawling task is divided into task types and processed by the corresponding sub-crawlers during the web content crawling process, the possibility of blocking can be reduced during large-scale web content crawling.

[0055] It needs to be further explained that, such as Figure 5 As shown in the embodiments of this application, the web crawler 2, in addition to including a main crawler 21, a task distribution platform 22, and multiple sub-crawlers 23, may also include a resource scheduling platform 24. The resource scheduling platform 24 is communicatively connected to each sub-crawler 23, enabling data communication between the resource scheduling platform 24 and the task distribution platform 22. The resource scheduling platform 24 can determine the task types to be processed by each sub-crawler 23 and send the task types to be processed by each sub-crawler 23 to the task distribution platform 22.

[0056] Therefore, the web content crawling method provided in this application embodiment may further include: a resource scheduling platform obtaining the usage information of different types of resources in each sub-crawler, and then determining the task type to be processed by each sub-crawler based on the usage information of different types of resources in each sub-crawler.

[0057] Each sub-crawler contains multiple types of resources, including CPU resources, memory resources, disk resources, network bandwidth resources, and physical address resources.

[0058] To determine the usage information of a sub-crawler's CPU resources, memory resources, disk resources, and physical address resources, you can use files such as / proc / stat and / proc / meminfo, or basic Linux commands such as htop and iostat. For example, you can use these commands to determine CPU utilization and CPU load as information on the sub-crawler's CPU resource usage; or you can use these commands to determine physical memory utilization, virtual memory utilization, and memory consumption percentage as information on the sub-crawler's memory resource usage; or you can use these commands to determine disk utilization, disk throughput (MB / s), and disk IOPS as information on the sub-crawler's disk resource usage; or you can use these commands to determine inode occupancy as information on the sub-crawler's physical address resource usage.

[0059] To determine the network bandwidth usage information of a sub-crawler, one can use files under / proc / net / and related basic Linux commands. For example, the commands mentioned above can be used to determine the connection latency between the sub-crawler and the main node, heartbeat checks, the network card's data flow and number of data packets received or sent per minute, the number of TCP connections, etc., as information on the network bandwidth usage of the sub-crawler.

[0060] For example, the sub-crawler can use the commands mentioned above to determine its own usage information for different types of resources, and then periodically report it to the resource scheduling platform 24. This allows the resource scheduling platform 24 to obtain the usage information for different types of resources periodically reported by each sub-crawler. The reporting period can be 1 hour, 0.5 hours, or other time periods.

[0061] After obtaining the usage information of different types of resources in each sub-crawler, the resource scheduling platform 24 can determine the task type to be processed by each sub-crawler based on the usage information of different types of resources in each sub-crawler. For example, if a sub-crawler has a high CPU utilization rate (CPU resource usage information) and a low virtual memory utilization rate (memory resource usage information), it can be determined that the task type to be processed by the sub-crawler is a memory resource intensive task, so that the sub-crawler can be used to process memory resource intensive tasks in the future; or if a sub-crawler has a low CPU utilization rate and a high virtual memory utilization rate, it can be determined that the task type to be processed by the sub-crawler is a CPU resource intensive task, so that the sub-crawler can be used to process CPU resource intensive tasks in the future.

[0062] In addition, combined Figure 6 The web crawler 2 shown above, this web crawler 2 in the above Figure 5 Based on the web crawler shown, a message queue bus 25 is further included. This message queue bus 25 is located between the main crawler 21 and the task distribution platform 22, and connects the main crawler 21 and the task distribution platform 22 respectively. The main crawler 21 can communicate with the task distribution platform 22 via the message queue bus 25.

[0063] For example, in step S43 above, the main crawler 21 sends the target task type of the network content crawling task to the task distribution platform 22. This can be achieved by the main crawler 21 sending the target task type of the network content crawling task to the task distribution platform 22 through the message queue bus 25. Specifically, the main crawler 21 sends the target task type of the network content crawling task to the message queue bus 25, which then forwards it to the task distribution platform 22. In this way, when the main crawler 21 needs to send multiple target task types of different network content crawling tasks to the task distribution platform 22, it can forward them through the message queue bus 25. This allows for the adjustment of the interval and frequency through the message queue bus 25, reducing the computational pressure on the task distribution platform 22.

[0064] Based on the same inventive concept as the web content crawling method provided in the embodiments of this application, the embodiments of this application also provide a web content crawling device. For any unclear aspects of this device embodiment, please refer to the corresponding content of the method embodiment. In particular, combined with... Figure 2 , Figure 5 and Figure 6 As shown, a web crawler includes a main crawler, a task distribution platform, and multiple sub-crawlers for handling different task types. This device is applied to web crawlers, as shown in Figure 7. The device 70 includes a sending unit 701 and a distribution unit 702, wherein:

[0065] Sending unit 701 is used to send the target task type of the web content crawling task to the task distribution platform, wherein the target task type is determined according to the task information carried in the task processing request of the web content crawling task.

[0066] The allocation unit 702 is used to allocate the web content crawling task to the corresponding sub-crawler for processing according to the target task type.

[0067] The device 70 provided in the embodiments of this application adopts the same inventive concept as the web content crawling method provided in the embodiments of this application. Under the premise that the method can solve the technical problem, the device 70 can also solve the technical problem, which will not be described in detail here.

[0068] In addition, in practical applications, the technical effects achieved by combining the device 70 with specific hardware and software, cloud technology, etc., are also within the scope of protection of this application. For example, using a distributed cluster approach to deploy different units in the device 70 in different nodes of the distributed cluster can improve efficiency.

[0069] The web crawler also includes a resource scheduling platform; and the device 70 may further include: an acquisition unit and a determination unit, wherein:

[0070] The acquisition unit is used to acquire information on the usage of different types of resources in each sub-crawler;

[0071] The determination unit is used to determine the type of task to be processed by each sub-crawler based on the usage information of different types of resources in each sub-crawler.

[0072] The resource types include CPU resources, memory resources, disk resources, network bandwidth resources, and physical address resources; and the task types include CPU resource-intensive tasks, memory resource-intensive tasks, disk resource-intensive tasks, network bandwidth resource-intensive tasks, and physical address resource-intensive tasks.

[0073] The acquisition unit may also include an acquisition subunit, which is used by the resource scheduling platform to acquire the usage information of different types of resources periodically reported by each sub-crawler.

[0074] The device 70 may further include a task processing request acquisition unit for acquiring a task processing request for the web content crawling task, wherein the task processing request carries the task information.

[0075] The task processing request can be entered by the user on the page displayed on the terminal; or, the task processing request can be generated by the triggering of a scheduled task.

[0076] The sending unit 701 may specifically include a sending subunit, which is used to send the target task type of the web content crawling task to the task distribution platform via a message queue bus.

[0077] Specifically, the main crawler can include: a main crawler based on the Node.js architecture.

[0078] This invention also provides a storage medium, comprising: a program, which, when run on an electronic device, enables the electronic device to execute all or part of the processes of the methods described in the above embodiments. The storage medium may be a disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc. The storage medium may also include combinations of the above types of memory.

[0079] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for crawling web content, characterized in that, A web crawler consists of a main crawler, a task distribution platform, and multiple sub-crawlers for handling different task types, including: The main crawler sends the target task type of the web content crawling task to the task distribution platform. The target task type is determined based on the task information carried in the task processing request of the web content crawling task. The task information includes: the network address, network channel, and task type classification label of the web content to be crawled. The task distribution platform assigns the web content crawling task to the corresponding sub-crawler for processing based on the target task type; The web crawler also includes a resource scheduling platform; and the method further includes: The resource scheduling platform obtains the usage information of different types of resources periodically reported by each sub-crawler; The resource scheduling platform determines the task type to be processed by each sub-crawler based on the usage information of different types of resources in each sub-crawler. The task types include CPU-intensive tasks and memory-intensive tasks; the resource types include CPU resources and memory resources; the usage information of different resource types includes CPU utilization and virtual memory utilization; the resource scheduling platform determines the task type to be processed by each sub-crawler based on the usage information of different resource types in each sub-crawler, including: If the sub-crawler has a high CPU usage but a low virtual memory usage, then the task type that the sub-crawler is used to process is a memory-intensive task. If the sub-crawler has low CPU usage but high virtual memory usage, then the task type that the sub-crawler is used to process is determined to be a CPU resource-intensive task.

2. The method according to claim 1, characterized in that, Resource types also include disk resources, network bandwidth resources, and physical address resources; and, Task types also include disk resource intensive tasks, network bandwidth resource intensive tasks, and physical address resource intensive tasks.

3. The method according to claim 1, characterized in that, The method further includes: The main crawler obtains the task processing request for the web content crawling task, and the task processing request carries the task information.

4. The method according to claim 3, characterized in that, The task processing request is entered by the user on the page displayed on the terminal; or, Task processing requests are generated by triggering scheduled tasks.

5. The method according to claim 1, characterized in that, The main crawler sends the target task type of the web content crawling task to the task distribution platform, specifically including: The main crawler sends the target task type of the web content crawling task to the task distribution platform through the message queue bus.

6. The method according to claim 1, characterized in that, The main crawler specifically includes: a main crawler based on the Node.js architecture.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for performing the method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, Includes: a program, when run on an electronic device, that enables the electronic device to perform the method as described in any one of claims 1 to 6.

Images