The invention relates to a Chinese webpage
data acquisition and
processing system and method, the
system comprises an acquisition module, a
data transmission module, a
data processing module, a scheduling module and a data storage module, and the method comprises the following steps: the scheduling module puts seed url / url of sites with different weights into a task
queue; the acquisition module reads the url to be acquired in the task
queue, accesses internet data by initiating a network request, and writes responded webpage data into a kafka
message queue in the
data transmission module in a structured manner; the data storage module generates a corresponding jsonl.gz file from the webpage data through the data in the consumption
data transmission module and uploads the file to s3 for storage; and the
data processing module reads the
html data in the s3 storage through spark, and carries out url extraction,
pagerank calculation and combined deduplication
processing. Compared with the prior art, large-scale and high-quality Chinese
data acquisition processing and storage can be realized, and the method has the advantages of high acquisition efficiency, accurate duplicate removal, high timeliness, flexible configuration and the like.