Chinese webpage data collecting and processing system and method thereof

Abstract

The invention relates to a Chinese webpage data acquisition and processing system and method, the system comprises an acquisition module, a data transmission module, a data processing module, a scheduling module and a data storage module, and the method comprises the following steps: the scheduling module puts seed url / url of sites with different weights into a task queue; the acquisition module reads the url to be acquired in the task queue, accesses internet data by initiating a network request, and writes responded webpage data into a kafka message queue in the data transmission module in a structured manner; the data storage module generates a corresponding jsonl.gz file from the webpage data through the data in the consumption data transmission module and uploads the file to s3 for storage; and the data processing module reads the html data in the s3 storage through spark, and carries out url extraction, pagerank calculation and combined deduplication processing. Compared with the prior art, large-scale and high-quality Chinese data acquisition processing and storage can be realized, and the method has the advantages of high acquisition efficiency, accurate duplicate removal, high timeliness, flexible configuration and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of web page data collection, and in particular to a Chinese web page data collection and processing system and method. Background Art

[0002] Currently, there are various web page collection technologies and constructed web page datasets. For example, Common Crawl is a platform for multilingual data collection from web pages. Common Crawl regularly crawls website data around the world through a large-scale distributed data collection system and stores it in a publicly accessible database. The goal of this project is to build a large-scale dataset containing all public web pages for researchers, developers, and the public to explore and analyze.

[0003] All crawled data of Common Crawl is open to the public for free, without registration or license application. The project contains hundreds of billions of pages and several PB of data, including images, videos, JSON files and other types of data, covering many websites around the world, but the collection of Chinese website data is not comprehensive. Common Crawl uses a distributed computing framework, Apache Hadoop computing framework and Apache Nutch data collection framework, based on the Java development language, which is difficult to develop, deploy and maintain; Common Crawl is based on batch URL (Uniform Resource Locator) deduplication, and there is no stock URL and page content deduplication. In addition, Common Crawl periodically collects pages and stores these data in a searchable database for update. Usually a new snapshot is released every few months, which may lack the latest data and lack data real-time. In addition, the data of Common Crawl is compressed in warc format, which requires special tools such as WAT files and WARC files to read, resulting in a lack of convenience in actual operation.

[0004] In summary, the existing technology lacks web page collection for Chinese data and cannot perform flexible and high-quality real-time data collection for large-scale data. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a Chinese web page data collection and processing system and method thereof, which can realize large-scale, high-quality Chinese data collection, processing and storage for Chinese websites.

[0006] The object of the present invention can be achieved by the following technical scheme: A Chinese web page data collection and processing system includes a collection module, a data transmission module, a data processing module, a scheduling module and a data storage module. The collection module writes the response web page data structure into a Kafka message queue in the data transmission module by accessing Internet data according to the URL to be collected;

[0007] The data storage module generates a corresponding jsonl.gz file from the web page data by consuming the data in the data transmission module and uploads it to s3 storage;

[0008] The data processing module is used to read web page data from S3 storage, extract the current URL and external links, calculate the pagerank value of each page in the batch, calculate the weight of each site domain name, and use the deduplication method of request URL + response URL + HTML feature combination, and use Spark to dedupe the batch and stock URLs as a whole;

[0009] The scheduling module automatically directs the requested tasks to different priority queues according to the weights of different site domain names, and the queues include site URLs with different weights.

[0010] A method for collecting and processing Chinese web page data comprises the following steps:

[0011] S1, the scheduling module puts the seed URL / site URLs of different weights into the task queue;

[0012] S2, the collection module reads the URL to be collected in the task queue, initiates a network request to access Internet data, and writes the response web page data structure into the Kafka message queue in the data transmission module;

[0013] S3, the data storage module consumes the data in the data transmission module, generates the corresponding jsonl.gz file from the web page data and uploads it to s3 storage;

[0014] S4, the data processing module reads the HTML data stored in S3 through Spark, and performs URL extraction, pagerank calculation, and combined deduplication processing.

[0015] Furthermore, the scheduling module in step S1 specifically evaluates the importance of each site or its influence on the Internet to dynamically adjust the processing order: assigning high-weight sites to higher-priority queues to ensure that requests from these important sites receive faster responses; assigning low-weight sites to lower-priority queues, thereby reasonably allocating resources without affecting overall performance.

[0016] Furthermore, in step S1, the scheduling module performs hash calculation on the site domain name to evenly distribute the requests to each queue, thereby ensuring that the data in the queue is more discrete.

[0017] Furthermore, the acquisition module in step S2 adopts a crawler architecture that combines multi-threading with multi-process concurrent requests, and the crawler architecture has built-in domestic and foreign proxy IP pools, automatic encoding trial units and memory control units.

[0018] Furthermore, in step S2, the data transmission module uses the Kafka message queue to construct the data transmission layer, and customizes the message body format of success and failure. When the data is successfully sent to the target system, a message body containing a success flag is generated;

[0019] For messages that cannot be processed successfully, a failure message body is created.

[0020] Furthermore, the domestic and foreign proxy IP pools are used to automatically adapt to the access requirements of different websites;

[0021] The automatic encoding trial unit is used to intelligently identify and convert the correct character set according to the web page content;

[0022] The memory control unit is used to release objects that are no longer in use and process a large amount of data streams using a generator mode.

[0023] Furthermore, the data storage module in step S3 specifically applies the GZIP compression algorithm based on JSONL, and generates a corresponding jsonl.gz file from the web page data by compressing it row by row.

[0024] Furthermore, the process of URL extraction and PageRank calculation in step S4 is as follows: first, the downloaded web page data is read, and the current URL and all external links are extracted therefrom. Then, the distributed computing capability of Spark is utilized to batch process the extracted data, and the weights of different site domain names are counted by calculating the PageRank value within each page.

[0025] Furthermore, the process of combined deduplication processing in step S4 is: adopting a deduplication method combining request url+response url+html features, and using spark to deduplicate the entire batch and stock URLs.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The Chinese web page data collection and processing system designed by the present invention includes a collection module, a data transmission module, a data processing module, a scheduling module and a data storage module. It adopts a modular structure and has the advantages of multi-module decoupling, high throughput and high system availability. It can realize large-scale, high-quality Chinese data collection, processing and storage.

[0028] The present invention collects Chinese web pages specifically, can identify the language type of web page texts and perform targeted filtering, and covers more comprehensive and complete Chinese web pages.

[0029] The present invention uses a distributed computing framework, an Apache Spark computing framework and a self-developed data acquisition framework, based on a Python development system, with low difficulty in development, deployment and maintenance.

[0030] The present invention uses Spark distributed computing, which makes deduplication and extraction more efficient and can perform parallel incremental deduplication during large-scale collection.

[0031] The present invention removes duplicates based on URL and page features, selects representative features in HTML pages for deduplication, and the deduplication results are more accurate.

[0032] The present invention compresses the data results into the jsonl.gz format, and the json compressed by row is more flexible and convenient to use.

[0033] The present invention automatically calculates the site weights and automatically schedules to give priority to collecting and updating sites frequently, thereby achieving better timeliness.

[0034] The present invention uses a random URL queue to reduce the access pressure on a certain site at the same time by disrupting the queue, while improving the proxy utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0036] Figure 2 It is a schematic diagram of the method flow of the present invention;

[0037] Figure 3 A schematic diagram of the system architecture constructed in the embodiment;

[0038] Figure 4 A schematic diagram of the scheduling module in the present invention automatically allocating priority queues according to site weights;

[0039] Figure 5 A schematic diagram of a hash request task in a scheduling module in the present invention;

[0040] Figure 6 This is a schematic diagram of the multi-process + multi-threaded acquisition architecture of the acquisition module in the present invention;

[0041] Description of the marks in the figure: 1. Acquisition module, 2. Data transmission module, 3. Data processing module, 4. Scheduling module, 5. Data storage module. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Example

[0044] like Figure 1 As shown, a Chinese web page data collection and processing system includes a collection module 1, a data transmission module 2, a data processing module 3, a scheduling module 4 and a data storage module 5, wherein the input end of the collection module 1 is connected to the scheduling module 4, the output end of the collection module 1 is connected to the data transmission module 2, the data transmission module 2 is connected to the data storage module 5, the data storage module 5 is connected to the data processing module 3, and the data processing module 3 is connected to the scheduling module 4, wherein the collection module 1 writes the corresponding web page data structured into the Kafka message queue in the data transmission module 2 according to the URL to be collected by accessing the Internet data;

[0045] The data storage module 5 consumes the data in the data transmission module 2, generates the corresponding jsonl.gz file from the web page data and uploads it to s3 storage;

[0046] Data processing module 3 is used to read web page data from S3 storage, extract the current URL and external links, calculate the pagerank value of each page in the batch, count the weight of each site domain name, and use the deduplication method of request URL + response URL + HTML feature combination, and use Spark to dedupe the batch and stock URLs as a whole;

[0047] The scheduling module 4 automatically directs the pending request tasks to different priority queues according to the weights of different site domain names, and the queues include site URLs of different weights.

[0048] Based on the above system, a Chinese web page data collection and processing method is implemented, such as Figure 2 As shown, the following steps are included:

[0049] S1, the scheduling module puts the seed URL / site URLs of different weights into the task queue;

[0050] S2, the collection module reads the URL to be collected in the task queue, initiates a network request to access Internet data, and writes the response web page data structure into the Kafka message queue in the data transmission module;

[0051] S3, the data storage module consumes the data in the data transmission module, generates the corresponding jsonl.gz file from the web page data and uploads it to s3 storage;

[0052] S4, the data processing module reads the HTML data stored in S3 through Spark, and performs URL extraction, pagerank calculation, and combined deduplication processing.

[0053] This embodiment applies the above solution to construct Figure 3 The system architecture shown includes:

[0054] Acquisition module - multi-thread + multi-thread concurrent request mode, built-in domestic and foreign proxy IP automatic adaptation, automatic encoding trial, memory control.

[0055] Data transmission module - Use the Kafka message queue to customize the success and failure message bodies, improve data stability and avoid the risk of data loss, and centrally monitor data status while achieving high throughput.

[0056] Data processing module - uses Apache Spark, a fast general-purpose computing engine for large-scale data processing.

[0057] Data processing - URL extraction: read the downloaded HTML data, extract the current URL and external links, calculate the pagerank value of each page in the batch, and count the weight of each site domain name;

[0058] Deduplication: Design a deduplication method that combines the request URL, response URL, and HTML features, and use Spark to deduplicate the entire batch and existing URLs.

[0059] Scheduling module - according to the weight of different site domain names, they enter different priority queues, and through the hash algorithm, the data in the queue is discretized, reducing the access pressure to a certain site and improving the proxy utilization rate.

[0060] Data storage module - uses row-compressed jsonl.gz format to reduce data volume and effectively save storage space. It uses s3 storage to ensure data reliability, security and low cost.

[0061] The workflow of the above system architecture is:

[0062] The scheduling module puts the seed URL / site URLs of different weights into the task queue;

[0063] The acquisition module reads the URL in the queue, initiates a network request, and writes the response web page data structured into the Kafka message queue;

[0064] The data storage module consumes the data in the data transmission module, generates a jsonl.gz file and uploads it to s3 storage;

[0065] The data processing module uses Spark to read HTML data stored on S3, extract the current URL and external links, calculate the pagerank value of each page in the batch, and count the weight of each site domain name. It uses the deduplication method of the request URL + response URL + HTML feature combination, and uses Spark to deduplicate the batch and existing URLs as a whole.

[0066] The following is a detailed description of each module in this solution:

[0067] 1. Scheduling Module

[0068] According to the calculated weights of different site domain names, an intelligent allocation mechanism is designed, which can automatically direct pending requests to different priority queues (such as Figure 4 The core of this design is to dynamically adjust the processing order by evaluating the importance of each site or its influence on the Internet: high-weight sites will be assigned to a higher priority queue to ensure that requests from these important sites are responded to faster; low-weight sites will be directed to a lower priority queue, thereby reasonably allocating resources without affecting overall performance.

[0069] In order to further optimize this process and avoid excessive access pressure on a single site, this solution also introduces a hash algorithm. By performing hash calculations on the site domain name, it can effectively distribute requests evenly to each queue (such as Figure 5 As shown in the figure, the data in the queue is more discrete. This not only reduces the pressure on a single site, but also improves the utilization of the entire proxy system, allowing resources to more efficiently serve various request processing tasks with different needs and priorities.

[0070] 2. Collection Module

[0071] like Figure 6As shown in the figure, the design of the crawler with Python multi-threading combined with multi-process concurrent request collection fully utilizes the multi-core advantages of modern computers and the network bandwidth capacity to achieve efficient data capture and processing procedures. This crawler architecture automatically adapts to the access requirements of different websites by means of built-in domestic and foreign proxy IP pools. At the same time, the crawler uses multi-threading technology to initiate HTTP requests in parallel, which significantly improves the speed and efficiency of data collection; and the addition of multiple processes further breaks through the performance bottleneck of a single thread, especially when dealing with large-scale data collection tasks, it can make better use of system resources. In order to better adapt to the content encoding of different websites, the crawler has a built-in automatic encoding trial function, which can intelligently identify and convert the correct character set according to the content of the web page, effectively solving the problem of data parsing errors caused by encoding mismatch. In addition, while achieving high performance, the crawler also pays special attention to memory management. Through refined memory control strategies, such as timely release of unused objects and the use of generator mode to process large amounts of data streams, it effectively avoids the risk of memory leaks or excessive resource consumption, ensuring that the crawler can maintain good stability and performance after long-term operation.

[0072] 3. Data transmission module

[0073] Using Kafka message queues to build a data transmission layer can effectively improve data stability and reduce the risk of data loss. In order to ensure the reliability of data transmission, this solution customizes the message body format of success and failure. In this way, when the data is successfully sent to the target system, a message body containing a success flag will be generated, which may also include relevant metadata, such as message ID, sending timestamp, etc., for tracking and auditing. For messages that are not successfully processed, a failure message body will be created, which records the cause of the failure, the number of retries, and other necessary contextual information, which helps to quickly locate the problem and take remedial measures.

[0074] With this design, Kafka, as a distributed stream processing platform, can not only support high-throughput data transmission, but also ensure that data can be persistently stored until successfully delivered even in the event of network partitions or system failures. In addition, through the integrated monitoring tools, the status of the entire message queue can be centrally monitored, including key indicators such as message production rate, consumption rate, latency, and failure rate. This not only allows you to grasp the health of the system in real time, but also allows you to respond quickly to any abnormal situations to ensure the smooth operation of the data processing pipeline. Overall, this method not only improves the efficiency and reliability of data processing, but also enhances the maintainability and scalability of the system, so that the data flow can more robustly support business needs.

[0075] 4. Data processing module

[0076] Using Apache Spark, a fast and general computing engine for large-scale data processing, massive data sets can be processed efficiently. In a specific application scenario, the downloaded HTML data is first read and the current URL and all external links are extracted from it. Then, using Spark's distributed computing capabilities, these data are batch processed and the weight of different site domain names is evaluated by calculating the PageRank value within each page. This PageRank-based statistical method can effectively reflect the importance of web pages, and then help identify which websites have a higher influence in the entire network.

[0077] In order to further optimize the processing flow, this solution also proposes a unique deduplication method that combines the characteristics of request URLs, response URLs, and HTML documents for matching. In this way, it is possible to correctly identify duplicate data even when the content is similar but the URLs are different. By leveraging the power of Spark, not only can the URLs in the current batch be deduplicated, but the URLs in historical data can also be further processed to ensure the uniqueness and accuracy of the entire data set. This method greatly improves the efficiency of data processing and reduces the waste of resources caused by duplicate data, ultimately achieving effective management and analysis of large-scale web crawler data.

[0078] 5. Data Storage Module

[0079] Using the line-compressed jsonl.gz format can significantly reduce the volume of data by about 90%, effectively saving storage space. JSONL (JSON Lines) is a lightweight data exchange format that allows each line to store an independent JSON record. This format is very suitable for processing large-scale log files or streaming data. By applying the GZIP compression algorithm on the basis of JSONL, the file size can be further reduced, which is especially beneficial for large data sets that need to be frequently uploaded and downloaded. This compression method can not only reduce storage costs, but also speed up data transmission, thereby improving the efficiency of the entire system. In order to ensure the reliability, security and low-cost storage of data, this solution uses Amazon S3 as the main storage solution. S3 provides a highly scalable object storage service with excellent persistence and availability characteristics, which can meet the needs of large-scale data storage.

[0080] To verify the effectiveness of this solution, this embodiment was put into use. Under the condition of no competition for proxy and other resources, the collection efficiency was 241 million pages / day (2800 pages / second). The number of collected pages reached 11.6 billion pages, and the number of mined websites reached 36.16 million websites.

[0081] In summary, the present invention designs a large-scale, high-quality Chinese data collection method for Chinese websites. The prior art lacks a method for scheduling large-scale data collection task queues. The present invention uses spark analysis weights and redis queues to design an efficient scheduling system, saving costs and improving efficiency.

[0082] The prior art lacks a method for deduplication of large-scale data collection increments. The present invention utilizes Spark deduplication to design a method that supports rapid deduplication of massive data.

[0083] The prior art lacks a specific function to determine whether it is a Chinese website. The present invention designs a distributed data acquisition system that can perform specific analysis on a page to determine whether it is a target web page (whether it is a Chinese website), and has the ability to discover new websites and add them to the scheduling system.

[0084] The prior art rarely involves efficient concurrent storage of data collection result pages. The present invention designs a page parsing storage system that can store data collection result pages with high concurrency.

[0085] The system configuration of the present invention is more flexible. The present invention can flexibly configure domestic and foreign proxy options to adapt to different network environments, flexibly control concurrency, proxy, storage and other links, and design a management interface to control the efficiency and speed of collection.

[0086] Compared with Common Crawl, which has serious data loss of Chinese web pages, the present invention can automatically identify the language of page content, and can more accurately limit the scope of collection, and the covered Chinese web pages are more comprehensive and complete. The present invention is based on the python development system, and compared with the Common Crawl based on java, the difficulty of development, deployment, maintenance and other links is less; the present invention uses Spark distributed computing, and compared with the Apache Hadoop computing framework used by Common Crawl, the deduplication and extraction efficiency are more efficient; the present invention is based on url and page feature deduplication, and compared with Common Crawl based on batch url deduplication, the deduplication result of the present invention is more accurate; the present invention compresses the data results into jsonl.gz format, and the json compressed by line is more flexible and convenient to use, and does not need to use a special reader to read the results; the present invention automatically calculates the site weight, and automatically schedules the priority collection of frequently updated sites, which is better in timeliness than the method of publishing a new snapshot every once in a while; the present invention uses a random url queue, and through the disruption of the queue, the access pressure to a certain site at the same time is reduced, and the agent utilization rate is improved at the same time.

Claims

1. A Chinese web page data collection and processing system, characterized in that: The method comprises a collection module (1), a data transmission module (2), a data processing module (3), a scheduling module (4) and a data storage module (5); the collection module (1) accesses Internet data according to the URL to be collected, and writes the corresponding web page data into a Kafka message queue in the data transmission module (2) in a structured manner; The data storage module (5) generates a corresponding jsonl.gz file from the web page data by consuming the data in the data transmission module (2) and uploads it to s3 storage; The data processing module (3) is used to read web page data from S3 storage, extract the current URL and external links therefrom, calculate the pagerank value of each page in the batch, calculate the weight of each site domain name, and adopt a deduplication method combining request URL+response URL+html features, and use Spark to dedupe the batch and stock URLs as a whole; The scheduling module (4) automatically directs the requested tasks to different priority queues according to the weights of different site domain names, wherein the queues include site URLs with different weights.

2. A Chinese web page data collection and processing method, applied to a Chinese web page data collection and processing system as claimed in claim 1, characterized in that: The following steps are involved: S1, the scheduling module puts the seed URL / site URLs of different weights into the task queue; S2, the collection module reads the URL to be collected in the task queue, initiates a network request to access Internet data, and writes the response web page data structure into the Kafka message queue in the data transmission module; S3, the data storage module consumes the data in the data transmission module, generates the corresponding jsonl.gz file from the web page data and uploads it to s3 storage; S4, the data processing module reads the HTML data stored in S3 through Spark, and performs URL extraction, pagerank calculation, and combined deduplication processing.

3. A Chinese web page data collection and processing method according to claim 2, characterized in that: The scheduling module in step S1 specifically evaluates the importance of each site or its influence on the Internet to dynamically adjust the processing order: assign high-weight sites to higher-priority queues to ensure that requests from these important sites receive faster responses; assign low-weight sites to lower-priority queues, thereby reasonably allocating resources without affecting overall performance.

4. A Chinese web page data collection and processing method according to claim 2, characterized in that: In step S1, the scheduling module performs hash calculation on the site domain name to evenly distribute the requests to each queue, ensuring that the data in the queue is more discrete.

5. A Chinese web page data collection and processing method according to claim 2, characterized in that: The acquisition module in step S2 adopts a crawler architecture that combines multi-threading with multi-process concurrent requests. The crawler architecture has built-in domestic and foreign proxy IP pools, an automatic encoding trial unit, and a memory control unit.

6. A Chinese web page data collection and processing method according to claim 2, characterized in that: In step S2, the data transmission module uses the Kafka message queue to build a data transmission layer and customizes the message body format of success and failure. When the data is successfully sent to the target system, a message body containing a success flag is generated; For messages that cannot be processed successfully, a failure message body is created.

7. A Chinese web page data collection and processing method according to claim 5, characterized in that: The domestic and foreign proxy IP pools are used to automatically adapt to the access requirements of different websites; The automatic encoding heuristic unit is used to intelligently identify and convert the correct character set according to the webpage content; The memory control unit is used to release objects that are no longer used and process a large amount of data streams using a generator mode.

8. A Chinese web page data collection and processing method according to claim 2, characterized in that: The data storage module in step S3 specifically applies the GZIP compression algorithm based on JSONL, and generates a corresponding jsonl.gz file from the web page data by compressing it row by row.

9. A Chinese web page data collection and processing method according to claim 2, characterized in that: The process of URL extraction and PageRank calculation in step S4 is as follows: first, the downloaded web page data is read, and the current URL and all external links are extracted therefrom. Then, the extracted data is batch processed using the distributed computing capability of Spark, and the weights of different site domain names are counted by calculating the PageRank value within each page.

10. A Chinese web page data collection and processing method according to claim 2, characterized in that: The process of combined deduplication processing in step S4 is: adopting a deduplication method combining request url+response url+html features, and using spark to deduplicate the entire batch and stock URLs.

Images