Web site product detailed information classification crawling and product information base establishing method

Abstract

The invention discloses a webpage crawling method for Web site product information obtaining. First, a home page of first-level classification of site products is subjected to crawling, and by analyzing crawled classification home page source files, a next-level product classification home page link is obtained; then step-by-step crawling is carried out until all classification home pages of a site are subjected to crawling; and by analyzing source files of all classification sub-pages, page turning elements and classification page numbers are obtained, then classification sub-page links are generated, and finally, according to the classification sub-page links, classification sub-page crawling is completed. Meanwhile, by analyzing crawled product classification sub-page source files, product detailed information and classification information of products are extracted, the mapping relation of site product id, classification id and other detailed information is established, and a product information base is established.

Description

technical field [0001] The invention relates to the field of Internet web crawlers. For websites with product numbers, use web crawlers to establish mapping relationships between product ids, category ids, and other detailed information. Background technique [0002] With the rapid development of Internet technology, the continuous enrichment of Web site products, and the continuous improvement of people's awareness of the value of information, thus stimulating people's demand for mining useful information from the massive product information of Web sites. Accurately classifying massive product information and establishing a product information database is an important basis for mining useful information. In the entire process of data mining, web crawlers play an important role. The web pages captured by web crawlers are the data sources for big data analysis. These data will directly affect the accuracy of data mining. Comprehensive crawling of page information, such a l...

AI Technical Summary

Problems solved by technology

In the entire process of data mining, web crawlers play an important role. The web pages captured by web crawlers are the data sources for big data analysis. These data will directly affect the accuracy of data mining. However, traditional web crawlers try to Comprehensive crawling of page information, such a lack of purposeful crawling will lead to a reduction in the efficiency of traditional crawlers. This method is not suitable for websites containing massive product information

Images