A URL cleaning system and method based on integrated learning

Abstract

The invention discloses a URL cleaning system and method based on integrated learning, which crawls the URL of the website and its corresponding website title; judges whether the website title is consistent with the specified crawling theme; The title of the website is marked as a category A title, otherwise it is marked as a category B title; the marked A category title and B category title are word-segmented, and the naive Bayesian algorithm is trained and predicted according to the word segmentation results, and the corresponding word segmentation results are constructed. Then use the Stacking algorithm to perform fusion processing to obtain the fusion result; finally use the decision tree algorithm to train and predict the fusion result to obtain a decision tree model, and clean the URL through the decision tree model; thus extremely It greatly improves the URL cleaning efficiency, saves a lot of manual inspection time, and improves the accuracy of verifying the website title corresponding to the URL.

Description

technical field [0001] The invention relates to the technical field of network information processing, in particular to an integrated learning-based URL cleaning system and a method for applying the system. Background technique [0002] URL, also known as a web page address, is a standard resource address on the Internet, and is used to completely describe an identification method for the addresses of web pages and other resources on the Internet. Every web page on the Internet has a unique URL address name identification, usually called a URL address, this address can be a local disk, or a computer on the LAN, and more often a site on the Internet . Simply put, a URL is a web address, commonly known as a "web address". Usually crawler engineers need to clean the data after crawling the website data, and the more troublesome thing is the cleaning of the URL. [0003] The title of the website can be said to be the face of the company. The traditional system is easy to miss...

AI Technical Summary

Problems solved by technology

[0003] The title of the website can be said to be the face of the company. The traditional system is easy to miss strange titles, such as: "Angel in White", which can be clearly seen as the website of the hospital, but this title is likely to be filtered out by the traditional system

Therefore, the disadvantage of the traditional system is that it needs to manually label the data, which will consume a certain amount of manpower in the early stage

Images