Webpage data acquisition method based on fast and slow thread pool
By adopting fast and slow thread pool separation task processing in web page data collection, the problem of low download efficiency in the existing technology is solved, efficient massive data processing and resource utilization are achieved, and data download speed and stability are improved.
Patent Information
- Application Number
- CN202410860699.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-15
AI Technical Summary
The existing multi-threading technology cannot fully utilize the system resource performance in massive web page data collection, resulting in low download efficiency and inability to process data volumes of millions or even tens of millions.
Using web page data collection method based on fast and slow thread pool, the task is separated by fast thread pool and slow thread pool. The fast thread pool handles tasks with fast response speed, and the slow thread pool handles tasks with slow response speed, and realizes task timeout or failure Retry and log saving, dynamically adjust the thread pool size to adapt to data volume changes.
Make full use of system resources to avoid overload system load and network congestion, realize ultra-high-speed web page data download, improve download efficiency, and save time and costs.
Smart Images

Figure CN118963928A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of web page data collection, and in particular to a web page data collection method based on a fast and slow thread pool. Background Art
[0002] In the massive web page data collection, the efficiency of using single thread collection is low, so most of them will use multithreading to collect web page data. Common multithreading technology can download multiple web pages at the same time, thereby increasing the download speed, and parallel downloading can make full use of system resources and network bandwidth; in addition, you can also use special tools and libraries to speed up the download speed, such as wget, curl, Scrapy and other tools can simplify the development process, and provide some optimization functions, such as automatic retry, breakpoint resume, concurrency management, etc.
[0003] Although these methods improve the speed of web page downloads, they cannot fully utilize the performance of system resources to achieve high-performance data download capabilities, nor can they process millions or even tens of millions of data simultaneously. Summary of the invention
[0004] The present invention provides a web page data collection method, device, equipment and storage medium based on a fast and slow thread pool.
[0005] According to the first aspect of the present disclosure, a method for collecting web page data based on a fast and slow thread pool is provided. The method comprises:
[0006] Get the list of URLs to be processed, and extract the URLs in the list of URLs to be processed;
[0007] Traverse the download tasks executed by each sub-thread in the fast thread pool, move the completed tasks to the completed task list, and call the sub-thread in the fast thread pool to download the extracted URL;
[0008] Traverse the download tasks executed by each sub-thread in the slow thread pool, move all the traversed tasks to the completed task list, and call the sub-thread in the slow thread pool to execute the tasks that have timed out or failed to be executed by each sub-thread in the fast thread pool;
[0009] Traverse the tasks in the completed task list, and save the task information of the completed tasks in the completed task list to the log file;
[0010] If there are still URLs to be downloaded in the pending URL list, the URLs in the pending URL list are extracted repeatedly, and the operations on the fast thread pool, the slow thread pool and the completed task list are performed in sequence until all URLs in the pending URL list are extracted.
[0011] In some implementations of the first aspect, the method further includes:
[0012] When a child thread in the fast thread pool is called to download an extracted URL, if there are URLs to be downloaded in the extracted URLs, if the number of URLs to be downloaded is greater than the number of idle child threads in the fast thread pool, and the number of threads in the fast thread pool is less than the maximum number of threads, a new child thread is created in the fast thread pool to download the URLs to be downloaded in the extracted URLs.
[0013] In some implementations of the first aspect, the task information includes:
[0014] The task ID, the URL corresponding to the task, the task start time, the task end time, and the task status;
[0015] The task status includes task failure, task timeout, and task success.
[0016] In some implementations of the first aspect, the child thread executes the download task, including:
[0017] The child thread initiates a data collection request to the corresponding web page according to the corresponding URL, and captures the abnormal situation of the data collection request;
[0018] If no abnormal data collection request is captured, the web page responds to the corresponding data collection request, and the child thread obtains and stores the header information and HTML information of the web page;
[0019] The status of the download task corresponding to the data collection request responded by the web page is updated to task completion, and the task information of the task is recorded.
[0020] In some implementations of the first aspect, the child thread executes the download task, further comprising:
[0021] If the data collection request times out, the status of the download task corresponding to the timeout exception request is updated to task timeout, and the task information of the task is recorded.
[0022] In some implementations of the first aspect, the method further includes:
[0023] Traverse the tasks in the completed task list, and save the timed-out or failed tasks in the completed task list to the pending URL list;
[0024] If the download task executed by the child thread is still not completed within the preset number of repetitions or the preset repetition time, the task information of the task is saved to the log file, and the task is no longer saved to the list of pending URLs.
[0025] According to a second aspect of the present disclosure, a web page data collection device based on a fast and slow thread pool is provided. The device comprises:
[0026] The task list acquisition module is used to obtain a list of URLs to be processed and extract URLs from the list of URLs to be processed;
[0027] The fast thread pool calling module is used to traverse the download tasks executed by each sub-thread in the fast thread pool, move the completed tasks to the completed task list, and call the sub-thread in the fast thread pool to download the extracted URL;
[0028] The slow thread pool calling module is used to traverse the download tasks executed by each sub-thread in the slow thread pool, move all the tasks obtained by the traversal to the completed task list, and call the sub-threads in the slow thread pool to execute the tasks that have timed out or failed to be executed by each sub-thread in the fast thread pool;
[0029] The completed task list calling module is used to traverse the tasks in the completed task list and save the task information of the completed tasks in the completed task list to a log file;
[0030] The web page data collection loop module is used to repeatedly extract the URLs in the list of pending URLs, execute operations on the fast thread pool, the slow thread pool and the completion task list in sequence if there are still URLs to be downloaded in the list of pending URLs, until all the URLs in the list of pending URLs are extracted.
[0031] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.
[0032] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.
[0033] In the present disclosure, the download tasks executed by each sub-thread in the fast thread pool are traversed, the completed tasks are moved to the completed task list, and the sub-thread in the fast thread pool is called to download the URL extracted from the list of pending URLs; the download tasks executed by each sub-thread in the slow thread pool are traversed, all the tasks obtained by the traversal are moved to the completed task list, and the sub-thread in the slow thread pool is called to execute the tasks that have timed out or failed to execute by each sub-thread in the fast thread pool; the tasks in the completed task list are traversed, and the task information of the completed tasks in the completed task list is saved to the log file; if there are still URLs to be downloaded in the list of pending URLs, the above steps are repeated in sequence until all URLs in the list of pending URLs are extracted. In this way, the performance of system resources can be fully utilized, the system overload and network congestion can be avoided, and ultra-high-speed web page data download can be achieved, thereby improving download efficiency and saving time and cost.
[0034] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:
[0036] Figure 1 A flowchart of a web page data collection method based on a fast and slow thread pool provided by an embodiment of the present disclosure is shown;
[0037] Figure 2 A structural diagram of a web page data collection device based on a fast and slow thread pool provided by an embodiment of the present disclosure is shown;
[0038] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0040] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0041] In view of the problems in the background technology, the disclosed embodiment provides a web page data collection method and device based on a fast and slow thread pool. Specifically, the download tasks executed by each sub-thread in the fast thread pool are traversed, the completed tasks are moved to the completed task list, and the sub-thread in the fast thread pool is called to download the URL extracted from the list of pending URLs; the download tasks executed by each sub-thread in the slow thread pool are traversed, all the tasks obtained by the traversal are moved to the completed task list, and the sub-thread in the slow thread pool is called to execute the tasks that have timed out or failed to execute by each sub-thread in the fast thread pool; the tasks in the completed task list are traversed, and the task information of the completed tasks in the completed task list is saved to a log file; if there are still URLs to be downloaded in the list of pending URLs, the above steps are repeated in sequence until all URLs in the list of pending URLs are extracted. In this way, the performance of system resources can be fully utilized, the system overload and network congestion can be avoided, and ultra-high-speed web page data download can be achieved, thereby improving download efficiency and saving time and cost.
[0042] The web page data collection method and device based on the fast and slow thread pools provided by the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings through specific embodiments.
[0043] Figure 1 A flowchart of a webpage data collection method based on a fast and slow thread pool provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps:
[0044] S110, obtaining a list of URLs to be processed, and extracting URLs from the list of URLs to be processed.
[0045] In some embodiments, the read_csv function is called to obtain the URL list to be processed from the URL list file path to be processed.
[0046] S120, traversing the download tasks executed by each sub-thread in the fast thread pool, moving the completed tasks to the completed task list, and calling the sub-thread in the fast thread pool to download the extracted URL.
[0047] In some embodiments, the number of child threads in the fast thread pool is a preset value.
[0048] In some embodiments, the tasks executed by the fast thread pool are tasks with a faster response speed. Concurrently executing multiple tasks with a faster response speed can effectively utilize network bandwidth and system resources and increase the speed of downloading web page data.
[0049] In some embodiments, tasks that have timed out or failed to execute in each sub-thread in the fast thread pool may also be moved to a slow task list.
[0050] In some embodiments, the method 100 further includes:
[0051] When a child thread in the fast thread pool is called to download an extracted URL, if there are URLs to be downloaded in the extracted URLs, if the number of URLs to be downloaded is greater than the number of idle child threads in the fast thread pool, and the number of threads in the fast thread pool is less than the maximum number of threads, a new child thread is created in the fast thread pool to download the URLs to be downloaded in the extracted URLs.
[0052] In some embodiments, calling a child thread in a fast thread pool to download the extracted URL includes:
[0053] Call the child threads in the fast thread pool to download the extracted URL so that each child thread can perform the download task;
[0054] Specifically, a subthread in the fast thread pool is called to download the extracted URL, a task object corresponding to the downloaded URL is created, and the download task corresponding to the corresponding task object is executed; wherein,
[0055] The task object is a DownloadTask object; the DownloadTask object includes task information.
[0056] In some embodiments, if each sub-thread in the fast thread pool performs a download task for the first time, that is, the web page data collection cycle is the first cycle, the sub-thread in the fast thread pool is called to download the extracted URL so that each sub-thread performs the download task;
[0057] Otherwise, traverse the download tasks executed by each child thread in the fast thread pool, move the completed tasks in the previous cycle to the completed task list, and call the child thread in the fast thread pool to execute the timed out or failed tasks in the completed task list in the previous cycle.
[0058] In some embodiments, when creating a task object, a constructor (such as an _int_ function) receives command line parameters passed externally, and when the child thread executes the download task, it is determined whether the task is completed based on the information in the command line parameters; wherein,
[0059] The command line parameters include URL information corresponding to the download task executed by the child thread and a data collection request time threshold, etc.; wherein the data collection request time threshold is used to determine whether the data collection request has timed out.
[0060] In some embodiments, the task information includes:
[0061] The task ID, the URL corresponding to the task, the task start time, the task end time, and the task status; the task status includes task failure, task timeout, and task success.
[0062] In some embodiments, the child thread performs a download task, including:
[0063] The child thread initiates a data collection request to the corresponding web page according to the corresponding URL, and captures the abnormal situation of the data collection request;
[0064] If no abnormal data collection request is captured, the web page responds to the corresponding data collection request, and the child thread obtains and stores the header information and HTML information of the web page;
[0065] The status of the download task corresponding to the data collection request responded by the web page is updated to task completion, and the task information of the task is recorded.
[0066] In some embodiments, the done attribute of a task with a task status of completed is true.
[0067] In some embodiments, the child thread initiates a data collection request to the corresponding web page according to the corresponding URL, and captures an abnormal situation of the data collection request, including:
[0068] The child thread calls the web_collection function to collect web page data;
[0069] Specifically, the child thread uses the requests.get function to initiate a data collection request to the corresponding web page according to the corresponding URL, and uses the try function and the except function to capture the abnormal situation of the data collection request.
[0070] In some embodiments, the child thread obtains and stores the header information and HTML information of the web page, including:
[0071] If there is a data storage directory, the header information in the obtained web page is stored in a txt file in the data storage directory, and the HTML information in the obtained web page is stored in an HTML file in the data storage directory;
[0072] If the data storage directory does not exist, the child thread creates a new storage directory according to the current date, and stores the header information in the obtained web page to a txt file in the newly created storage directory, and stores the HTML information in the obtained web page to an HTML file in the newly created storage directory.
[0073] In some embodiments, the child thread executes the download task, further comprising:
[0074] If the data collection request times out, the status of the download task corresponding to the timeout exception request is updated to task timeout, and the task information of the task is recorded.
[0075] In some embodiments, the child thread executes the download task, further comprising:
[0076] If other exceptions occur in the data collection request, the status of the download task corresponding to the other exception request is updated to task failure, and the task information of the task is recorded.
[0077] In some embodiments, each sub-thread executes a task. After the sub-thread completes the corresponding task, it can continue to execute other tasks, or it can terminate the sub-thread and create a new sub-thread to execute other tasks.
[0078] S130, traversing the download tasks executed by each sub-thread in the slow thread pool, moving all the tasks obtained by the traversal to the completed task list, and calling the sub-threads in the slow thread pool to execute the tasks that have timed out or failed to be executed by each sub-thread in the fast thread pool.
[0079] In some embodiments, there is no limit on the number of child threads in the slow thread pool.
[0080] In some embodiments, the tasks performed by the slow thread pool are tasks with a slower response speed, such as downloading large files, complex pages, or data in pages that need to be parsed and processed. By executing these tasks with a slower response speed through the slow thread pool, it can be ensured that these tasks are processed without taking up too many system resources.
[0081] In some embodiments, a child thread in a slow thread pool is called to execute a timed-out or failed task in a slow task list.
[0082] In some embodiments, if each sub-thread in the slow thread pool executes the download task for the first time, that is, the webpage data collection cycle is the first cycle, the sub-thread in the slow thread pool is called to execute the task that each sub-thread in the fast thread pool executes timed out or failed in this cycle;
[0083] Otherwise, traverse the download tasks executed by each child thread in the slow thread pool, move all the tasks in the previous cycle obtained by traversal to the completed task list, and call the child threads in the slow thread pool to execute the tasks that timed out or failed in this cycle and the previous cycle.
[0084] S140, traversing the tasks in the completed task list, and saving the task information of the completed tasks in the completed task list to a log file.
[0085] In some embodiments, the method 100 further includes:
[0086] Traverse the tasks in the completed task list, and save the timed-out or failed tasks in the completed task list to the pending URL list;
[0087] If the download task executed by the child thread is still not completed within the preset number of repetitions or the preset repetition time, the task information of the task is saved to the log file, and the task is no longer saved to the list of pending URLs.
[0088] S150, if there are still URLs to be downloaded in the list of pending URLs, then the URLs in the list of pending URLs are extracted repeatedly, and operations on the fast thread pool, the slow thread pool and the completed task list are performed in sequence until all URLs in the list of pending URLs are extracted.
[0089] In some embodiments, the steps in method 100 are executed by a main thread, the run function is used to define the execution method of the main thread and / or each sub-thread, and the start function is used to start the main thread and / or each sub-thread.
[0090] In some embodiments, before each repeated extraction of URLs in the list of pending URLs, execution of operations on the fast thread pool, the slow thread pool, and the completion of the task list, the main thread will sleep for a certain period of time, such as 0.1 seconds, to avoid problems such as network congestion caused by frequent operations.
[0091] In some embodiments, the number of threads of the fast thread pool is set by an externally passed command line parameter.
[0092] According to the embodiment of the present disclosure, by traversing the download tasks executed by each sub-thread in the fast thread pool, the completed tasks are moved to the completed task list, and the sub-thread in the fast thread pool is called to download the URL extracted from the list of pending URLs; traversing the download tasks executed by each sub-thread in the slow thread pool, all the tasks obtained by traversal are moved to the completed task list, and the sub-thread in the slow thread pool is called to execute the tasks that have timed out or failed to execute by each sub-thread in the fast thread pool; traversing the tasks in the completed task list, the task information of the completed tasks in the completed task list is saved to the log file; if there are still URLs to be downloaded in the list of pending URLs, the above steps are repeated in sequence until all URLs in the list of pending URLs are extracted. In this way, the performance of system resources can be fully utilized, the system overload and network congestion can be avoided, and ultra-high-speed web page data download can be achieved, thereby improving download efficiency and saving time and cost.
[0093] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0094] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0095] Figure 2 The structure diagram of a web page data collection device based on a fast and slow thread pool provided by an embodiment of the present disclosure is shown. The device 200 includes:
[0096] The task list acquisition module 210 is used to acquire a list of URLs to be processed and extract URLs from the list of URLs to be processed.
[0097] The fast thread pool calling module 220 is used to traverse the download tasks executed by each sub-thread in the fast thread pool, move the completed tasks to the completed task list, and call the sub-thread in the fast thread pool to download the extracted URL.
[0098] In some embodiments, the apparatus 200 is further configured to:
[0099] When a child thread in the fast thread pool is called to download an extracted URL, if there are URLs to be downloaded in the extracted URLs, if the number of URLs to be downloaded is greater than the number of idle child threads in the fast thread pool, and the number of threads in the fast thread pool is less than the maximum number of threads, a new child thread is created in the fast thread pool to download the URLs to be downloaded in the extracted URLs.
[0100] In some embodiments, module 220 is specifically configured to:
[0101] The child thread performs download tasks, including:
[0102] The child thread initiates a data collection request to the corresponding web page according to the corresponding URL, and captures the abnormal situation of the data collection request;
[0103] If no abnormal data collection request is captured, the web page responds to the corresponding data collection request, and the child thread obtains and stores the header information and HTML information of the web page;
[0104] The status of the download task corresponding to the data collection request responded by the web page is updated to task completion, and the task information of the task is recorded.
[0105] In some embodiments, module 220 is further configured to:
[0106] The child thread performs the download task, which also includes:
[0107] If the data collection request times out, the status of the download task corresponding to the timeout exception request is updated to task timeout, and the task information of the task is recorded.
[0108] In some embodiments, module 220 is further configured to:
[0109] Task information includes:
[0110] The task ID, the URL corresponding to the task, the task start time, the task end time, and the task status;
[0111] The task status includes task failure, task timeout, and task success.
[0112] The slow thread pool calling module 230 is used to traverse the download tasks executed by each sub-thread in the slow thread pool, move all the traversed tasks to the completed task list, and call the sub-threads in the slow thread pool to execute the tasks that have timed out or failed to execute by each sub-thread in the fast thread pool.
[0113] The completed task list calling module 240 is used to traverse the tasks in the completed task list and save the task information of the completed tasks in the completed task list to a log file.
[0114] In some embodiments, module 240 is specifically configured to:
[0115] Traverse the tasks in the completed task list, and save the timed-out or failed tasks in the completed task list to the pending URL list;
[0116] If the download task executed by the child thread is still not completed within the preset number of repetitions or the preset repetition time, the task information of the task is saved to the log file, and the task is no longer saved to the list of pending URLs.
[0117] The web page data collection loop module 250 is used to repeatedly extract the URLs in the list of URLs to be processed, execute operations on the fast thread pool, the slow thread pool and the completion task list in sequence if there are still URLs to be downloaded in the list of URLs to be processed, until all URLs in the list of URLs to be processed are extracted.
[0118] Understandably, Figure 2 Each module / unit in the illustrated device 200 has the function of implementing each step in the detection method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect, which will not be described in detail here for the sake of brevity.
[0119] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0120] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.
[0121] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0122] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).
[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0124] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0126] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.
[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A web page data collection method based on fast and slow thread pools, characterized in that: include: Get the list of URLs to be processed, and extract the URLs in the list of URLs to be processed; Traversing the download tasks executed by each sub-thread in the fast thread pool, moving the completed tasks to the completed task list, and calling the sub-thread in the fast thread pool to download the extracted URL; Traverse the download tasks executed by each sub-thread in the slow thread pool, move all the traversed tasks to the completed task list, and call the sub-thread in the slow thread pool to execute the tasks that have timed out or failed to be executed by each sub-thread in the fast thread pool; Traverse the tasks in the completed task list, and save the task information of the completed tasks in the completed task list to the log file; If there are still URLs to be downloaded in the pending URL list, the URLs in the pending URL list are extracted repeatedly, and the operations on the fast thread pool, the slow thread pool and the completed task list are performed in sequence until all URLs in the pending URL list are extracted.
2. The method according to claim 1, characterized in that The method further comprises: When a child thread in the fast thread pool is called to download the extracted URL, when there are URLs to be downloaded in the extracted URLs, if the number of URLs to be downloaded is greater than the number of idle child threads in the fast thread pool, and the number of threads in the fast thread pool is less than the maximum number of threads, a new child thread is created in the fast thread pool to download the URLs to be downloaded in the extracted URLs.
3. The method according to claim 1, characterized in that: The task information includes: The task ID, the URL corresponding to the task, the task start time, the task end time, and the task status; The task status includes task failure, task timeout, and task success.
4. The method according to claim 1, characterized in that: The sub-thread executes the download task, including: The child thread initiates a data collection request to the corresponding web page according to the corresponding URL, and captures the abnormal situation of the data collection request; If no abnormal data collection request is captured, the web page responds to the corresponding data collection request, and the child thread obtains and stores the header information and HTML information of the web page; The status of the download task corresponding to the data collection request responded by the web page is updated to task completion, and the task information of the task is recorded.
5. The method according to claim 4, characterized in that The sub-thread executes the download task, and further includes: If the data collection request times out, the status of the download task corresponding to the timeout exception request is updated to task timeout, and the task information of the task is recorded.
6. The method according to claim 1, characterized in that The method further comprises: Traverse the tasks in the completed task list, and save the timed-out or failed tasks in the completed task list to the pending URL list; If the download task executed by the child thread is still not completed within the preset number of repetitions or the preset repetition time, the task information of the task is saved to the log file, and the task is no longer saved to the list of pending URLs.
7. A web page data collection device based on a fast and slow thread pool, characterized in that: include: The task list acquisition module is used to obtain a list of URLs to be processed and extract URLs from the list of URLs to be processed; A fast thread pool calling module is used to traverse the download tasks executed by each sub-thread in the fast thread pool, move the completed tasks to the completed task list, and call the sub-thread in the fast thread pool to download the extracted URL; The slow thread pool calling module is used to traverse the download tasks executed by each sub-thread in the slow thread pool, move all the tasks obtained by the traversal to the completed task list, and call the sub-threads in the slow thread pool to execute the tasks that have timed out or failed to be executed by each sub-thread in the fast thread pool; The completed task list calling module is used to traverse the tasks in the completed task list and save the task information of the completed tasks in the completed task list to a log file; The web page data collection loop module is used to repeatedly extract the URLs in the list of pending URLs, execute operations on the fast thread pool, the slow thread pool and the completion task list in sequence if there are still URLs to be downloaded in the list of pending URLs, until all the URLs in the list of pending URLs are extracted.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
