Automatic network data acquisition method

Abstract

The invention relates to the technical field of network data, and discloses an automatic network data acquisition method, which comprises the following steps: S1, acquiring network data to obtain an original webpage; s2, performing data extraction on the original webpage to obtain an analyzed webpage; s3, carrying out null removal, error removal, repetition removal, normalization and incomplete value supplement processing on the analyzed webpage to obtain processed data; s4, storing the processed data; and S5, processing the stored data. According to the automatic network data collection method, 24-hour uninterrupted collection can be carried out on data disclosed by a third platform, minute-level third-party platform data retrieval synchronization is supported, second-level updating can be achieved for data updating of increment parts of multiple sites, manual supervision is not needed, meanwhile, through keyword retrieval configuration, the data updating efficiency is improved, and the efficiency is improved. According to the method, irrelevant contents can be filtered out while automatic retrieval is realized, the accuracy is improved, and non-supervision, non-omission and rapid iterative data acquisition is realized.

Description

technical field [0001] The invention relates to the technical field of network data, in particular to an automatic network data collection method. Background technique [0002] Network data acquisition refers to the process of using Internet search engine technology to achieve targeted, industry-specific, and accurate data capture, and classify data according to certain rules and screening standards, and form a database file. [0003] At present, when acquiring data from 10 domestic and foreign industry sites, if we want to retrieve relevant content in a short time and update it to our site through traditional data collection and sorting methods, not only will the labor cost be large, but the retrieval speed will also be slow. The search frequency is also low, and the third-party platform data cannot be updated synchronously, and there may be errors in the manually collected data. Contents of the invention [0004] The purpose of the present invention is to provide an aut...

AI Technical Summary

Problems solved by technology

[0003] At present, when acquiring data from 10 domestic and foreign industry sites, if we want to retrieve relevant content in a short time and update it to our site through traditional data collection and sorting methods, not only will the labor cost be large, but the retrieval speed will also be slow. The search frequency is also low, and the third-party platform data cannot be updated synchronously, and there may be errors in the manually collected data

Images