Web crawling method, device, computing device and storage medium

Abstract

The present disclosure discloses a webpage crawling method, apparatus, computing device and storage medium, the method comprising: creating a crawling queue, obtaining a webpage address from a message queue and placing the webpage address into the crawling queue; starting a crawler process, and starting a browser object through the crawler process, wherein the crawler process is executed by at least one thread; controlling a coroutine corresponding to the thread to obtain the webpage address from the crawling queue, so as to initiate an access request using the webpage address, and suspending the coroutine when the access request initiated by the coroutine waits for a response, wherein the thread switches to execute an unsuspended coroutine to initiate a new access request or process the response returned by the request. According to the embodiments of the present disclosure, the processor and bandwidth utilization can be effectively improved, and the system resource consumption can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of crawler technology, and in particular to a webpage crawling method, apparatus, computing device, and storage medium. Background Art

[0002] A crawler is a program that uses computers' ability to quickly execute repetitive tasks to automatically retrieve web page information according to predefined rules. The earliest crawler technology relied on sending requests to retrieve the source code of static pages. Later, webkit-based crawlers emerged, using browsers to retrieve page data.

[0003] Current crawler technology faces two main challenges: First, network requests take a long time to complete, often idling the CPU even when multiple processes are waiting for requests to return. Second, launching a large number of requests simultaneously can easily cause network congestion, leading to connection timeouts and crawling failures. Summary of the Invention

[0004] The disclosure section introduces a series of simplified concepts that will be further described in detail in the specific embodiments section. The disclosure section of this disclosure is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0005] In view of the above technical problems, the present disclosure provides a web page crawling method, apparatus, computing device and storage medium, which can effectively improve processor and bandwidth utilization and reduce system resource consumption.

[0006] According to one aspect of the present disclosure, a webpage crawling method is provided, comprising:

[0007] Create a crawl queue, obtain the web page address from the message queue and put the web page address into the crawl queue;

[0008] Starting a crawler process to start a browser object through the crawler process, wherein the crawler process is executed by at least one thread;

[0009] The coroutine corresponding to the control thread obtains the web page address from the crawling queue to initiate an access request to the target page using the web page address, and suspends the coroutine when the access request initiated by the coroutine waits for a response. The thread switches to execute the unsuspended coroutine to initiate a new access request or process the response returned by the request.

[0010] According to another aspect of the present disclosure, there is provided a webpage crawling device, comprising:

[0011] A queue creation module is used to create a crawling queue, obtain a web page address from a message queue and put the web page address into the crawling queue;

[0012] A crawling startup module, configured to start a crawler process to start a browser object through the crawler process, wherein the crawler process is executed by at least one thread;

[0013] The crawling execution module is used to control the coroutine corresponding to the thread to obtain the web page address from the crawling queue to initiate an access request using the web page address, and to suspend the coroutine when the access request initiated by the coroutine is waiting for a response. The thread switches to execute the unsuspended coroutine to initiate a new access request or process the response returned by the request.

[0014] According to another aspect of the present disclosure, a computing device is provided, comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the web page crawling method according to one aspect of the present disclosure.

[0015] According to another aspect of the present disclosure, a computer storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the web page crawling method according to one aspect of the present disclosure is implemented.

[0016] According to the web crawling method and apparatus, computing device, and storage medium disclosed herein, during the web crawling process, a page access request is initiated by a coroutine corresponding to a thread, and the coroutine is suspended while the access request initiated by the coroutine is waiting for a response. The thread switches to executing an unsuspended coroutine to initiate a new access request. This allows the thread to initiate a new access request while the access request is waiting for a response, eliminating the need for the processor to wait for a response before initiating a new access request. This reduces waiting time and improves processor and bandwidth utilization. Furthermore, since page access requests are initiated by the coroutine corresponding to the thread, each crawling process can handle multiple access requests, changing the relationship between threads and access requests from one-to-one to one-to-many, thereby reducing the number of processes and system resource consumption while improving access request efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] FIG1 is a schematic flow chart of a webpage crawling method according to an embodiment of the present disclosure;

[0019] FIG2 is a schematic flow chart of a webpage crawling method according to an embodiment of the present disclosure;

[0020] FIG3 is a schematic flow chart of a webpage crawling method according to an embodiment of the present disclosure;

[0021] FIG4 is a schematic block diagram of a webpage crawling apparatus according to an embodiment of the present disclosure;

[0022] FIG5 is a schematic block diagram of a webpage crawling apparatus according to an embodiment of the present disclosure;

[0023] FIG6 is a schematic diagram of the structure of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present disclosure. However, it is obvious to those skilled in the art that the embodiments of the present disclosure can be implemented without one or more of these details. In other examples, in order to avoid confusion with the embodiments of the present disclosure, some technical features well known in the art are not described.

[0025] Unless otherwise specifically stated, the relative arrangement of the parts and steps, the numerical expressions and the numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure. Meanwhile, it should be understood that for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0026] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0027] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein. Based on the embodiments of the present disclosure described in this disclosure, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present disclosure.

[0028] FIG1 is a schematic flowchart of a webpage crawling method 100 according to an embodiment of the present disclosure.

[0029] Referring to FIG1 , a webpage crawling method 100 provided by an embodiment of the present disclosure includes:

[0030] Step S101, creating a crawling queue, obtaining a web page address from a message queue and placing the web page address into the crawling queue;

[0031] Step S102: starting a crawler process to start a browser object through the crawler process, wherein the crawler process is executed by at least one thread;

[0032] Step S103, control the coroutine corresponding to the thread to obtain the web page address from the crawling queue, use the web page address to initiate an access request to the target page, and suspend the coroutine when the access request initiated by the coroutine waits for a response, and the thread switches to execute the unsuspended coroutine to initiate a new access request or process the response returned by the request.

[0033] In step S101, the crawl queue is a local crawl queue of the crawler device or server. Each crawl queue can hold a predetermined number of web page addresses (e.g., web page URLs). For example, the crawl queue can hold 10 or more web page addresses. The crawler device's processor (CPU) can retrieve these web page addresses from a message queue using a single thread and place them into the crawl queue.

[0034] The message queue is a queue for all web page addresses (URLs) to be crawled. In one embodiment of the present disclosure, the message queue can be configured using Kafka (an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java). Web page addresses to be crawled (e.g., web page URLs) can be placed into the Kafka message queue. In S101, after creating the crawl queue, web page addresses can be retrieved from the Kafka message queue and placed into the crawl queue.

[0035] In step S102, a crawler process is started, and a browser object is started through the crawler process. The crawler process is executed by at least one thread. The crawler process is a process started by executing a crawler program. For each crawler device or server, multiple crawler processes can be started simultaneously.

[0036] For each started crawler process, each crawler process is executed by at least one thread of the processor of the crawler device. For example, the crawler process can be executed by one, two or more threads of the processor of the crawler device.

[0037] For each launched crawler process, each crawler process launches its own browser object. In one embodiment of the present disclosure, each crawler process launches one browser object. In other embodiments of the present disclosure, each crawler process may launch two or more browser objects. The browser object can use various suitable browsers as needed, such as commonly used browsers such as Chrome, Firefox, EDG, IE, 360, Sogou, Baidu, and QQ Browser.

[0038] In one embodiment of the present disclosure, the crawler process starts the browser object in headless mode to reduce system resource consumption for web crawling, such as reducing resource occupation and consumption of processors, memory, etc. Accordingly, in one embodiment of the present disclosure, the browser object preferably supports a browser in headless mode.

[0039] In one embodiment of the present disclosure, after the crawler process launches a browser object, the crawler process controls the browser object to launch multiple page objects, sets the size of the page objects, and performs identity disguise on the browser object. The page objects are allocated by the coroutine corresponding to the thread, and the identity disguise indicates the type of device initiating the access request. In other words, the browser object is controlled to open multiple blank pages for accessing target pages. The target page refers to the page corresponding to the webpage address, or the page opened via the webpage address.

[0040] In the above steps, the size of the page object is set according to needs, for example, according to the display size and resolution of the crawler device; for example, according to the web page crawling output requirements.

[0041] In the above steps, the browser object's identity disguise (i.e., UA disguise) is performed to disguise the crawler device according to the crawler's requirements. For example, if crawling mobile web pages requires it, the crawler device is disguised as a mobile device. Specifically, when initiating an access request based on the webpage address, the access request will include UA (User Agent) information. By disguising the identity, the target page or webpage will believe that the access request is initiated by the crawler's intended device. For example, if the UA disguise indicates that the accessing device is a mobile device, the target page will believe that the access request is initiated by a mobile device and will return a page configured for mobile devices. If the UA disguise indicates that the accessing device is a computer device, the target page will believe that the access request is initiated by a computer device and will return a page configured for computers.

[0042] In the above steps, the page object is allocated by the coroutine corresponding to the thread. That is, the page object is used by the coroutine corresponding to the thread to access the web page. In one embodiment of the present disclosure, the crawler process controls the browser object to launch multiple page objects, each of which is allocated by a coroutine.

[0043] In one embodiment of the present disclosure, starting a crawler process and starting a browser object through the crawler process may also include: executing a web page script to execute a crawler detection avoidance script to avoid being blocked by the crawler detection script when accessing the target page corresponding to the web page address.

[0044] In one embodiment of the present disclosure, the starting of the crawler process and the starting of the browser object through the crawler process may further include: executing redirection log output on the browser object to obtain output content of the browser object.

[0045] In step S103, the coroutine corresponding to the thread is controlled to obtain a web page address from the crawling queue, and then use the web page address to initiate an access request to the target page.

[0046] In one embodiment of the present disclosure, the coroutine is controlled to use the corresponding page object to initiate an access request for the acquired web page address. Specifically, each coroutine can allocate a page object started by the aforementioned browser object, and then use the page object to initiate an access request for the acquired web page address. Alternatively, each page object started by the aforementioned browser object has been assigned to a coroutine, and when the coroutine obtains the web page address, the page object corresponding to the coroutine (or the assigned page object) is used to initiate an access request for the acquired web page address. That is, each coroutine uses its own page object to initiate an access request for the web page address it obtains.

[0047] Furthermore, in step S103, the coroutine is suspended while the access request initiated by the coroutine is awaiting a response, and the thread switches to executing an unsuspended coroutine to initiate a new access request. Specifically, when an access request initiated by a coroutine enters the waiting phase (i.e., waiting for a response from the target page), the coroutine is suspended, and the thread to which it belongs switches to the main loop event, searching for the next unsuspended coroutine to execute. Executing an unsuspended coroutine can involve obtaining the webpage address from the unsuspended coroutine and initiating a new access request, or it can involve executing subsequent processing on the coroutine that obtained the response from the target page.

[0048] In this way, by suspending and switching coroutines, it is possible to avoid the processor being idle and wasting resources while waiting for access requests, thereby effectively improving processor and bandwidth utilization and reducing system resource consumption.

[0049] It should be understood that in the present disclosure, the number of threads assigned to execute the crawler process can be multiple, and each of the threads can have multiple coroutines. Accordingly, each coroutine will obtain the web page address from the crawling queue, and then use the page object to initiate an access request to the obtained web page address, and when the access request is waiting for a response, the thread switches to execute the unsuspended coroutine, which can avoid processor idleness due to waiting for the access request.

[0050] According to the web crawling method disclosed herein, during the web crawling process, a page access request is initiated by a coroutine corresponding to a thread, and the coroutine is suspended while the access request initiated by the coroutine is waiting for a response. The thread switches to executing an unsuspended coroutine to initiate a new access request. In this way, the thread can initiate a new access request while the access request is waiting for a response. This eliminates the need for the processor to wait for a request response before initiating a new access request, reducing waiting time and improving processor and bandwidth utilization. Furthermore, since page access requests are initiated by the coroutine corresponding to the thread, each crawling process can handle multiple access requests, changing the relationship between threads and access requests from one-to-one to one-to-many, thereby reducing the number of processes and system resource consumption while improving access request efficiency.

[0051] FIG2 is a schematic flowchart of a webpage crawling method 200 according to an embodiment of the present disclosure.

[0052] As shown in FIG2 , a webpage crawling method 200 provided by an embodiment of the present disclosure includes:

[0053] Step S201: Create a crawling queue, obtain a web page address from a message queue and put the web page address into the crawling queue;

[0054] Step S202: starting a crawler process to start a browser object through the crawler process, wherein the crawler process is executed by at least one thread;

[0055] Step S203: Control the coroutine corresponding to the thread to obtain the webpage address from the crawling queue, so as to initiate an access request to the target page using the webpage address, and suspend the coroutine when the access request initiated by the coroutine is waiting for a response. The thread switches to execute an unsuspended coroutine to initiate a new access request or process the response returned by the request;

[0056] Step S204, controlling the coroutine to render the target page when the initiated access request obtains a response to the target page;

[0057] Step S205: Save the rendered webpage code file of the target page and take a screenshot of the target page.

[0058] The above steps S201 to S203 are similar to steps S101 to S103 in the webpage crawling method 100 shown in FIG. 1 , and are not described in detail here.

[0059] In step S204, the target page enters a waiting state after initiating an access request. Once the target page responds, the coroutine is controlled to render the target page, for example, by parsing the target page components and filtering non-HTML addresses and blank pages. This can be done using a common rendering method, such as driving the mainstream Chromium rendering engine.

[0060] In one embodiment of the present disclosure, controlling the coroutine to render the target page when the initiated access request obtains a target page response includes:

[0061] The target page is detected, and if a pop-up window exists on the target page, the pop-up window is closed. That is, whether the target page has a pop-up window is detected, and if so, a close button is clicked to fully display the target page.

[0062] In the above step S205, after the rendering of the target page is completed, the webpage code of the target page, such as the HTML code, is saved, and a screenshot of the target page is taken, thereby completing the crawling of the target page.

[0063] It should be understood that for each co-routine, after completing the crawling of the acquired web page address, a new web page address is obtained from the crawling queue and a new access request is initiated. This process continues until all the web page addresses in the crawling queue are crawled. It should be understood that in this process, the crawling amount of each crawler program can be automatically balanced in the co-routine, thereby achieving load balancing and improving processor efficiency and bandwidth utilization.

[0064] This solution allows you to retrieve the code and screenshots of the target pages you crawl, thereby obtaining information about the target pages. Furthermore, it automatically detects and closes pop-up windows on each target page you visit, effectively increasing the amount of information captured in the screenshots. Furthermore, by driving the mainstream Chromium rendering engine to render the target pages, the success rate of capturing screenshots is significantly increased.

[0065] FIG3 is a schematic flowchart of a webpage crawling method 300 according to an embodiment of the present disclosure.

[0066] As shown in FIG2 , a webpage crawling method 300 provided by an embodiment of the present disclosure includes:

[0067] Step S301: Create a crawling queue, obtain a web page address from a message queue and put the web page address into the crawling queue;

[0068] Step S302: starting a crawler process to start a browser object through the crawler process, wherein the crawler process is executed by at least one thread;

[0069] Step S303: Control the coroutine corresponding to the thread to obtain the webpage address from the crawling queue, so as to initiate an access request to the target page using the webpage address, and suspend the coroutine when the access request initiated by the coroutine is waiting for a response. The thread switches to execute an unsuspended coroutine to initiate a new access request or process the response returned by the request;

[0070] Step S304: performing a timeout detection on the response of the target page. When the waiting time for the response of the target page reaches a preset waiting time threshold, the coroutine interrupts the access request initiated.

[0071] Step S305: Detect the timeout exception and perform corresponding processing according to the type of the detected timeout exception.

[0072] The above steps S301 to S303 are similar to steps S101 to S103 in the webpage crawling method 100 shown in FIG. 1 , and are not described in detail here.

[0073] In the above step S304, timeout detection of the response to the target page can be achieved by nesting the page request function into a custom timeout detection mechanism. When the waiting time for the response is longer than the timeout period set by the page or exceeds the maximum waiting time, the coroutine actively interrupts the access request operation and throws a timeout exception.

[0074] In the above step S305, the timeout exception thrown in step S304 is detected, and corresponding processing is performed according to the type of the detected timeout exception, that is, different types of timeout exceptions are processed in different ways.

[0075] In one embodiment of the present disclosure, the corresponding processing is performed according to the type of the detected timeout exception, specifically:

[0076] If the timeout exception is a connection timeout or a temporary domain name resolution anomaly, the access request is re-initiated, and the waiting time for the target page's response is extended based on the number of retries of the access request. Specifically, for timeout exceptions such as connection timeouts or temporary domain name resolution anomalies, the TCP (Transmission Control Protocol) congestion window mechanism can be used to extend the waiting time based on the number of retries of the access request after detecting the timeout exception, and the retry flag is decremented by 1. This recursive call to the crawling function is repeated to retry, thereby alleviating network congestion.

[0077] When the timeout exception is a connection rejection exception, the access request is re-initiated using the proxy IP. That is, for a connection rejection exception, the target page is crawled by trying to access using the proxy IP.

[0078] When the timeout exception is caused by an unresolvable domain name or a connection reset exception, the coroutine is controlled to obtain a new webpage address from the crawling queue to initiate a new access request. That is, for an unresolvable domain name or a connection reset exception, crawling of the current webpage address is terminated and crawling of the next webpage address from the crawling queue is started.

[0079] The above solution enables timeout and timeout exception detection. When the waiting time exceeds a set waiting time threshold, the access request is interrupted and, depending on the type of timeout exception, the access request is retried or the next web page address is accessed. This improves the crawler's success rate while reducing network congestion and unnecessary retries or waits. Specifically, by adding a custom timeout handling mechanism, the problem of long wait times for target page access requests is resolved, and by adding multiple retry mechanisms, the success rate is guaranteed while improving performance.

[0080] It should be understood that the web crawling method provided herein can be executed on one or more crawler devices. When multiple crawler devices are used, a distributed crawler can be implemented. When performing a distributed crawler, data distribution can be managed through a Kafka queue to achieve load balancing. Furthermore, new crawler devices can be added at any time during the distributed crawler process based on business needs.

[0081] FIG4 is a schematic structural block diagram of a webpage crawling device 400 according to an embodiment of the present disclosure.

[0082] The webpage crawling device 400 according to an embodiment of the present disclosure will be described below with reference to FIG. 4 .

[0083] 4 , a webpage crawling apparatus 400 according to an embodiment of the present disclosure includes a queue creation module 410 , a crawling initiation module 420 , a crawling execution module 430 , and a target page acquisition module 440 .

[0084] Queue creation module 410 is used to create a crawling queue, retrieve web page addresses from the message queue, and place them into the crawling queue. Queue creation module 410 is used to execute steps S101, S201, and S301 of the web crawling method described in conjunction with Figures 1-3. A detailed description of this process is provided in the preceding description of Figures 1-3 and will not be repeated here.

[0085] Crawling startup module 420 is used to initiate a crawler process, which is executed by at least one thread, to launch a browser object. Crawling startup module 420 is used to execute steps S102, S202, and S302 of the webpage crawling method described in conjunction with Figures 1-3. A detailed description of this process is provided in the description of Figures 1-3 and is not repeated here.

[0086] The crawling execution module 430 is used to control the coroutine corresponding to the thread to obtain the web page address from the crawling queue to use the web page address to initiate an access request, and suspend the coroutine when the access request initiated by the coroutine is waiting for a response, and the thread switches to execute the unsuspended coroutine. Figure 1 - Steps S103, S203, and S303 in the webpage crawling method described in FIG3 , for a detailed description of this process, please refer to the aforementioned description in conjunction with FIG1 to FIG3 , which will not be repeated here.

[0087] Target page acquisition module 440 is used to control the coroutine to render the target page when the initiated access request receives a response for the target page; save the rendered target page's webpage code file; and take a screenshot of the target page. Target page acquisition module 440 is used to execute steps S204-S205 of the webpage crawling method described in conjunction with FIG2 . A detailed description of this process is provided in the description of FIG2 above and is not repeated here.

[0088] Each module / unit in the web crawling device 400 shown in FIG4 has the function of implementing the corresponding steps in FIG1-3 and can achieve the corresponding technical effects. For the sake of brevity, they are not described here in detail.

[0089] FIG5 is a schematic structural diagram of a webpage crawling device 500 according to an embodiment of the present disclosure. Figure 5 A webpage crawling device 500 according to an embodiment of the present disclosure is described.

[0090] 5 , a webpage crawling apparatus 500 according to an embodiment of the present disclosure includes a queue creation module 510 , a crawling startup module 520 , a crawling execution module 530 , a timeout detection module 540 , and an exception handling module 550 .

[0091] Queue creation module 510 is used to create a crawling queue, retrieve web page addresses from the message queue, and place them into the crawling queue. Queue creation module 510 is used to execute steps S101, S201, and S301 of the web page crawling method described in conjunction with Figures 1-3. A detailed description of this process is provided in conjunction with Figures 1-3 and will not be repeated here.

[0092] The crawler initiation module 520 is used to initiate a crawler process, thereby launching a browser object through the crawler process. The crawler process is executed by at least one thread. The crawler initiation module 520 is used to execute steps S102, S202, and S302 of the webpage crawling method described in conjunction with Figures 1-3. A detailed description of this process is provided in the description of Figures 1-3 above and is not repeated here.

[0093] The crawling execution module 530 is used to control the coroutine corresponding to the thread to obtain the web page address from the crawling queue to use the web page address to initiate an access request, and suspend the coroutine when the access request initiated by the coroutine is waiting for a response, and the thread switches to execute the unsuspended coroutine. The crawling execution module 530 is used to execute the combined Figure 1- Steps S103, S203, and S303 in the webpage crawling method described in FIG3 , for a detailed description of this process, please refer to the aforementioned description in conjunction with FIG1 to FIG3 , which will not be repeated here.

[0094] Timeout detection module 540 is used to perform timeout detection on the response to the target page. When the waiting time for the response to the target page reaches a preset waiting time threshold, the access request initiated by the coroutine is terminated. Timeout detection module 540 is used to execute step S304 of the webpage crawling method described in conjunction with FIG3 . A detailed description of this process is provided in the description of FIG3 above and is not repeated here.

[0095] The exception handling module 550 is used to detect timeout exceptions and perform corresponding processing based on the type of the detected timeout exception. The exception handling module 550 is used to execute step S305 of the webpage crawling method described in conjunction with FIG3 . The detailed description of this process is referred to the description of FIG3 above and will not be repeated here.

[0096] Each module / unit in the web crawling device 500 shown in FIG5 has the function of implementing each step in FIG3 and can achieve its corresponding technical effect. For the sake of brevity, they are not described here in detail.

[0097] FIG6 shows a schematic diagram of the hardware structure of a computing device provided by an embodiment of the present disclosure.

[0098] The computing device 600 may include a processor 601 and a memory 602 storing computer program instructions.

[0099] Specifically, the processor 601 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0100] Memory 602 may include a large-capacity memory for data or instructions. By way of example, and not limitation, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be non-volatile solid-state memory. Memory 602 may be internal or external to the integrated gateway disaster recovery device.

[0101] In one example, the memory 602 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0102] The memory 602 may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0103] The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement the method / steps S101 to S103 in the embodiment shown in Figure 1, the method / steps S201 to S205 in the embodiment shown in Figure 2, and the method / steps S301 to S305 in the embodiment shown in Figure 2, and achieves the corresponding technical effects achieved by executing the methods / steps in the examples shown in Figures 1 to 3. For the sake of brevity, they are not repeated here.

[0104] The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement Figure 5The webpage crawling device 400 in the embodiment shown, as well as the queue creation module 410, the crawling start module 420, the crawling execution module 430 and the target page acquisition module 440, achieve the corresponding technical effects achieved by the device in the example shown in FIG4 , and implement the webpage crawling device 500, the queue creation module 510, the crawling start module 520, the crawling execution module 530, the timeout detection module 540 and the exception handling module 550, and achieve Figure 5 The corresponding technical effects achieved by the device in the illustrated example are not described in detail here for the sake of brevity.

[0105] In one example, the computing device 600 may further include a communication interface 603 and a bus 610. As shown in FIG6 , the processor 601, the memory 602, and the communication interface 603 are connected via the bus 610 and communicate with each other.

[0106] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present disclosure.

[0107] Bus 610, comprising hardware, software, or both, couples the components of the online data traffic metering device to one another. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although the present disclosure describes and illustrates specific buses, the present disclosure contemplates any suitable bus or interconnect.

[0108] The computing device 600 can execute the webpage crawling method in the embodiment of the present disclosure, thereby realizing the webpage crawling method described in conjunction with Figures 1-3.

[0109] Furthermore, according to an embodiment of the present disclosure, a storage medium is provided. Program instructions are stored on the storage medium. When executed by a computer or processor, the program instructions are used to execute the corresponding steps of the webpage crawling method according to the embodiment of the present disclosure and to implement the corresponding units or modules of the webpage crawling apparatus according to the embodiment of the present disclosure. The storage medium may include, for example, a personal computer hard disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the foregoing. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0110] In one embodiment, when the computer program instructions are executed by a computer, they can implement various functional modules in the webpage crawling device according to the embodiment of the present disclosure, and / or can execute the webpage crawling method according to the embodiment of the present disclosure.

[0111] In one embodiment, the computer program instructions perform the following steps when run by a computer: create a crawl queue, obtain a web page address from a message queue and place the web page address into the crawl queue; start a crawler process, and start a browser object through the crawler process, the crawler process being executed by at least one thread; control the coroutine corresponding to the thread to obtain the web page address from the crawl queue, so as to initiate an access request using the web page address, and suspend the coroutine when the access request initiated by the coroutine is waiting for a response, and the thread switches to execute the unsuspended coroutine.

[0112] According to the web crawling method and apparatus, computing device, and storage medium disclosed herein, during the web crawling process, a page access request is initiated by a coroutine corresponding to a thread, and the coroutine is suspended while the access request initiated by the coroutine is waiting for a response. The thread switches to executing an unsuspended coroutine to initiate a new access request. This allows the thread to initiate a new access request while the access request is waiting for a response, eliminating the need for the processor to wait for a response before initiating a new access request. This reduces waiting time and improves processor and bandwidth utilization. Furthermore, since page access requests are initiated by the coroutine corresponding to the thread, each crawling process can handle multiple access requests, changing the relationship between threads and access requests from one-to-one to one-to-many, thereby reducing the number of processes and system resource consumption while improving access request efficiency.

[0113] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present disclosure. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present disclosure. All such changes and modifications are intended to be included within the scope of the present disclosure as required by the appended claims.

[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0116] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0117] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various disclosed aspects, in the description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach of the present disclosure should not be interpreted as reflecting the intention that the claimed disclosure requires more features than are expressly recited in each claim. More precisely, as reflected in the corresponding claims, the point of disclosure is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present disclosure.

[0118] Those skilled in the art will understand that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings), as well as all processes or units of any method or apparatus disclosed herein, may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0119] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this disclosure and to form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0120] The various component embodiments of the present disclosure can be implemented in hardware, as software modules running on one or more processors, or as a combination thereof. Those skilled in the art will appreciate that, in practice, a microprocessor or digital signal processor (DSP) can be used to implement some or all of the functionality of some modules in the item analysis device according to the embodiments of the present disclosure. The present disclosure can also be implemented as a device program (e.g., a computer program or computer program product) for performing part or all of the methods described herein. Such a program implementing the present disclosure can be stored on a computer-readable medium or in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0121] It should be noted that the above embodiments illustrate rather than limit the present disclosure, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present disclosure may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0122] The foregoing descriptions are merely specific embodiments or illustrations of specific embodiments of the present disclosure, and the scope of protection of the present disclosure is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed within the scope of protection of the present disclosure. The scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A webpage crawling method, comprising: Create a local crawling queue to temporarily store web page addresses; Configure a message queue through Kafka, obtain the web page address from the message queue and put the web page address into the crawling queue; Initiating a crawler process, where the crawler process is executed by at least one thread, and launching a browser object through the crawler process in headless mode, specifically comprising: controlling the browser object with the crawler process to launch multiple page objects, setting the size of the page objects, and performing identity disguise on the browser object, wherein the page objects are allocated by the coroutine corresponding to the thread, and the identity disguise indicates the type of the device initiating the access request; Controlling the coroutine corresponding to the thread to obtain the web page address from the crawling queue, so as to use the web page address to initiate an access request to the target page through the page object, and perform a timeout detection on the response of the target page. When the response waiting time of the target page reaches a preset waiting time threshold, the coroutine interrupts the access request initiated and suspends the coroutine. The thread switches to execute the unsuspended coroutine to initiate a new access request or process the response returned by the request; Detect the timeout exception and perform corresponding processing according to the type of the detected timeout exception; the corresponding processing according to the type of the detected timeout exception includes: when the timeout exception is a connection timeout or a temporary domain name resolution exception, re-initiate the access request, and extend the response waiting time of the target page according to the number of retries of the access request; when the timeout exception is a connection rejection exception, use the proxy IP to re-initiate the access request; when the timeout exception is an unresolvable domain name or a connection reset exception, control the coroutine to obtain the new web page address from the crawling queue to initiate a new access request.

2. The method according to claim 1, wherein The starting of the crawler process and starting the browser object through the crawler process includes: Execute web scripts that evade crawler detection.

3. The method according to claim 1, wherein The starting of the crawler process and starting the browser object through the crawler process includes: Redirecting log output is performed on the browser object to obtain output content of the browser object.

4. The method according to any one of claims 1 to 3, wherein: Also includes: Controlling the coroutine to render the target page when the initiated access request obtains a response to the target page; The rendered webpage code file of the target page is saved and a screenshot of the target page is taken.

5. The method according to claim 4, wherein The controlling the coroutine to render the target page when the initiated access request obtains a target page response includes: The target page is detected, and when a pop-up window exists on the target page, the pop-up window is closed.

6. A webpage crawling device, comprising the method according to any one of claims 1 to 5, comprising: Queue creation module, used to create a local crawling queue for temporarily storing web page addresses; Configure a message queue through Kafka, obtain the web page address from the message queue and put the web page address into the crawling queue; A crawling startup module is configured to start a crawler process, the crawler process being executed by at least one thread, and to start a browser object through the crawler process in headless mode. Specifically, the crawler process controls the browser object to start multiple page objects, sets the size of the page objects, and performs identity disguise on the browser objects, wherein the page objects are allocated by the coroutine corresponding to the thread, and the identity disguise indicates the type of device initiating the access request. a crawling execution module, configured to control the coroutine corresponding to the thread to obtain the web page address from the crawling queue, to use the web page address to initiate an access request to a target page through a page object, to perform a timeout detection on the response of the target page, and when the response waiting time of the target page reaches a preset waiting time threshold, the coroutine interrupts the initiated access request and suspends the coroutine, and the thread switches to execute an unsuspended coroutine to initiate a new access request or process the response returned by the request; Detect the timeout exception and perform corresponding processing according to the type of the detected timeout exception; the corresponding processing according to the type of the detected timeout exception includes: when the timeout exception is a connection timeout or a temporary domain name resolution exception, re-initiate the access request, and extend the response waiting time of the target page according to the number of retries of the access request; when the timeout exception is a connection rejection exception, use the proxy IP to re-initiate the access request; when the timeout exception is an unresolvable domain name or a connection reset exception, control the coroutine to obtain the new web page address from the crawling queue to initiate a new access request.

7. A computing device, comprising: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the web page crawling method according to any one of claims 1 to 5.

8. A computer storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the webpage crawling method according to any one of claims 1 to 5.

Images