Self-adaption web crawler method based on machine learning

Abstract

The invention provides a self-adaption web crawler method based on machine learning. Because at present, software developers need to inquire about complex page codes and research rules to compile net crawler programs, no self-adaption programs exist to automatically guide crawlers to work and meanwhile the field of electronic commerce is not touched, the technical aim of mining a large number of data through the machine learning technology so as to position information related to the content on an unknown webpage needs to be achieved. The aim is achieved through the machine learning and data mining method. The method includes the main technical steps of data obtaining, characteristic extraction, isomerous data normalization, training data establishment, self-adaption training, learning method verification and self-adaption mode generation, the method is used for conducting position characteristic extraction and self-adaption training on the page codes of crawled websites, and a certain effect is achieved for research on self-adaption network crawlers in the field of electronic commerce.

Description

technical field [0001] The invention relates to computer technology, which mainly uses data mining and machine learning methods to solve the self-adaptive matching problem of web crawlers in the field of e-commerce, and belongs to the application fields of computer technology, data mining, machine learning and web crawler intersecting technologies. Background technique [0002] With the explosive growth of Internet information and the continuous rise of e-commerce websites, more and more people have begun to pay attention to how to find their favorite products in the e-commerce network with large data volume, and it is becoming more and more important for software industry professionals It is more inclined to do automatic processing in large amounts of data to improve the efficiency of information retrieval and achieve the purpose of targeted recommendation information. Crawlers, as the main way to obtain web page information, are also well known by more people. However, th...

AI Technical Summary

Problems solved by technology

However, the writing of web crawler programs at this stage usually requires professional software developers to query the page codes, study the corresponding rules, and also need to write different crawler programs for different websites and pages, and none of them can be self-adaptive. Functional program to automatically guide crawlers to crawl isomorphic information

At the same time, with the rise of major e-commerce companies, people in modern society are more inclined to online shopping, which has become an increasingly irresistible trend. How users can find the information they need in e-commerce websites with large amounts of data is a problem. very serious challenge

Images