Self-adaptive data acquisition method and device

Through the feature extraction and fusion processing of web page source code and crawler information, a crawler strategy generation model is generated, which solves the problem of insufficient dynamic adaptability of data acquisition methods in the existing technology, and realizes efficient and accurate data acquisition, reducing maintenance costs and improving collection quality.

CN120386907AActive Publication Date: 2025-07-29BEIJING SCI & TECH PATENT OFFICE
6 Cites 4 Cited by

Patent Information

Application Number
CN202510450850.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-04-11
Publication Date
2025-07-29
Estimated Expiration
2045-04-11

AI Technical Summary

Technical Problem

Existing data acquisition methods lack dynamic adaptability to target data sources, resulting in waste of resources, redundancy of data or omission of key information, especially in the collection of web text data, it is difficult to obtain the required information efficiently and accurately.

Method used

By obtaining the pending web page source code information, historical web page source code collection and crawler information collection, feature extraction and fusion processing are performed, crawler strategy generation model is generated, acquisition strategy is dynamically adjusted, the operation frequency, access depth and algorithm type of crawler program are optimized, and machine learning models are used for training and prediction, and finally an efficient acquisition strategy is generated.

Benefits of technology

Adaptive adjustment of the acquisition strategy is achieved according to actual conditions, improving the efficiency and quality of data acquisition, and reducing the daily maintenance costs of technicians.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a self-adaptive data acquisition method and device. The method comprises the following steps: acquiring to-be-processed webpage source code information, a historical webpage source code set and a crawler information set; the historical webpage source code set comprises a plurality of pieces of historical webpage source code information; the crawler information set comprises a plurality of pieces of crawler program information; performing fusion processing on the historical webpage source code set and the crawler information set to obtain crawler strategy generation model information; and generating model information by utilizing the crawler strategy, and processing the to-be-processed webpage source code information to obtain webpage collection information. Therefore, according to the embodiment of the invention, the acquisition strategy can be adaptively adjusted according to actual conditions, and data acquisition can be efficiently and accurately carried out, so that the daily maintenance cost of technicians can be reduced, and the efficiency and quality of data acquisition can be improved.
Need to check novelty before this filing date? Find Prior Art