Distributed acquisition method and system oriented to user generated content

Abstract

The invention provides a distributed collection method oriented to user generated content. The method includes the steps that first, page types are divided according to collection amount of collection pages and collection difficulty, and on the basis of the page types, collection tasks are built and added into a collection queue, wherein the collection tasks comprise composite collection tasks, and the composite collection tasks are collection tasks formed in the mode that a plurality of same type collection pages are classified into same collection tasks; second, the collection tasks are taken out of the collection queue concurrently, the collection tasks are executes, and acquired information is fed back. The invention further provides a corresponding distributed collection system which comprises a master control node and a plurality of sub-nodes, wherein the master control node is used for constructing and maintaining the queue of the collection tasks, and the sub-nodes are used for executing the collection tasks concurrently. According to the distributed collection method and system, collection speed is high, and the UGC news collection instantaneity is remarkably improved; the distributed collection method and system is applicable to collection of the pages in different types, and diversified collection tasks are executed; monitoring measures of collected objects can be avoided.

Description

technical field [0001] The present invention relates to the technical field of information collection, in particular, the present invention relates to a distributed collection method and system for content generated by users. Background technique [0002] User Generated Content is referred to as UGC (User Generated Content). UGC news is news event information uploaded or shared by users in social media (such as Weibo, blog, social network, etc.). UGC content has also become a major source of information for traditional media due to its timely response and fast dissemination. At present, with the popularization of Internet technology and the vigorous development of WEB2.0 technology, ordinary users have become the main producers of content on the Internet. However, due to the low threshold of UGC news, any user can upload content to the Internet, UGC news lacks effective supervision, and there are a lot of fake news. [0003] UGC-based news certification and early warning ...

AI Technical Summary

Problems solved by technology

[0004] From the perspective of technological progress at home and abroad, on the one hand, the existing distributed collection schemes do not focus on efficiency, but on continuity and stability. On the other hand, the current collection tasks focus on single-page collection, and each sub-node usually collects one page. This solution cannot meet the specific but diverse page collection requirements in news certification and early warning.

In short, the types of UGC information that need to be collected for news certification and early warning are complex and require high real-time performance. There is currently no corresponding data solution

Images