A sample training system based on idc harmful information monitoring system

Abstract

The invention discloses a sample training system based on an IDC harmful information monitoring system. In the crawler sample training unit, the topic correlation calculation module combines the webpage information captured by the crawler system with the crawler sample database to calculate the topic correlation of the webpage, and according to the Topic correlation adjusts the URL queue, filters out URLs below the preset threshold, and feeds back the calculated topic correlation value to the crawler sample training module. After the crawler sample training module performs training and learning, it updates the crawler sample database; Harmful monitoring samples In the training unit, the keyword approximate vocabulary training module, the harmful information monitoring system performs harmful detection according to the approximate vocabulary related to the input string generated by the approximate matching algorithm, and the keyword approximate vocabulary training module is determined according to the search result fitting degree calculation module The accuracy of search results, judging the similarity of similar words, and updating effective similar words to the harmful monitoring sample database.

Description

technical field [0001] The invention relates to a sample training system based on an IDC harmful information monitoring system. Background technique [0002] With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. As a tool to assist people in retrieving information, search engines become the entrance and guide for users to access the World Wide Web. However, these universal search engines also have certain limitations. [0003] Faced with an increasingly active network community environment, every netizen may become a publisher and disseminator of harmful information, and the channels of harmful communication on the Internet are becoming wider and wider, including blogs, news, forums, Weibo, and other channels. Web crawler is a pioneering technology that can be realized by various search engines. The advent of the era of big data ...

AI Technical Summary

Problems solved by technology

[0004] However, due to the limitations of traditional search engines, such as low coverage, poor timeliness, inaccurate results, and too many irrelevant results are gradually reflected.

The IDC system cannot accurately and effectively monitor harmful information

Images