Domain recognition plus topic recognition to build a machine learning model to detect dark links in web pages

Abstract

The invention provides a method for establishing a machine learning model to check webpage hidden links through domain identification and theme identification. The method comprises the steps of collecting a great number of webpage source codes which are marked as the webpage source codes comprising hidden links and the webpage source codes marked as the normal webpage source codes, and forming a training set based on the webpage source codes; extracting a risk text, risk degree, theme difference degree, a theme, a risk text vector, a risk text abnormal probability and the risk text length in each web page source code through suspicious domain identification, sensitive domain identification, secure domain identification, all-domain analysis and theme identification; carrying out model training on characteristic data of all web page source codes in the training set through utilization of a machine learning algorithm, thereby obtaining a classification distinguishing model; and importingthe characteristic data of to-be-predicted web page source codes into the classification distinguishing model, thereby obtaining a result of whether the to-be-predicted web page source codes comprisethe hidden links or not. According to the method, the highly mixed hidden link code identification effect is good, the feature extraction is relatively complete and the problem that a traditional method cannot accurately distinguish the hidden links and page tampering can be solved well.

Description

technical field [0001] The invention belongs to the technical field of network security, and in particular relates to a method for detecting dark links in webpages by constructing a machine learning model based on domain identification and topic identification. Background technique [0002] In recent years, the Internet industry has developed vigorously, and the Internet has become the main way for people to obtain information. With the emergence of various new websites, Internet information has grown exponentially. With this massive amount of information, search engines have become the main information search tools. Search engines crawl website information and calculate the weight of web page content to rank and display it in the search results. Since the websites displayed at the front of the search results have a higher probability of being visited by users, some website managers often resort to various cheating methods in order to obtain more visits. "Dark link" is a ch...

AI Technical Summary

Problems solved by technology

Only one of the currently published dark link detection patents uses a machine learning algorithm. The patent (application number 201410452221.2 publication number CN104239485A) uses machine learning to build a model to identify dark links based on all the anchor text extracted from the page as features. Extracting all the anchor text on the page will generate a lot of noise data, and the extracted features will be thicker, which will reduce the recognition effect, and because only the anchor text is used as a feature, the content of the page without hidden links will be tampered and misidentified as hidden links

Images