A quick rule customization method based on python QT and intelligent algorithm

Abstract

The invention discloses a quick rule customization method based on python QT and intelligent algorithm, which relates to the technical field of web page rule customization. The invention includes inputting URL of a page to be climbed, and loading the page by the client through the URL. extracting navigation list items based on selenium; extracting the body of detail page by intelligent algorithm;obtaining page element rules from page by js technology and returning to client; uploading the rules to the server, and the background crawler performing crawling according to the rules. The inventionextracts navigation list items in the page through selenium and filters out (a) labels whose ordinate is greater than the height of the browser and (a) labels whose abscissa is the same and whose number is less than the reference value. After that, the detailed page text is extracted by intelligent algorithm, which avoids the problem that the customized web page rules do not apply to many kinds of complicated web page rules by manual analysis, and is suitable for customizing web page rules with different websites, so as to improve the efficiency of customizing web page rules.

Description

technical field [0001] The invention belongs to the technical field of webpage rule customization, in particular to a fast rule customization method based on pythonQT and intelligent algorithms. Background technique [0002] With the rapid development of big data technology, data, as its fundamental research object, is playing an increasingly important role. How to obtain data efficiently and quickly has become one of the important topics of current research. As the basic technical means of Internet data acquisition, crawlers can efficiently acquire data and optimize and improve the current technology is imperative. At present, the basic idea for crawlers to obtain web page data is: [0003] (1) Through the given address of the target webpage, the crawler initiates a request for the page, that is, sends a Request, and the request can contain additional hearder and other information. [0004] (2) Obtain the content of the response after requesting the server. If the serve...

AI Technical Summary

Problems solved by technology

[0009] The purpose of the present invention is to provide a fast rule customization method based on pythonQT and intelligent algorithms, and extract the navigation list items in the page through selenium and filter out unconformable labels; meanwhile, extract the text part of the details page through intelligent algorithms, and realize multiple different websites. The customization of web page rules solves the problem that the existing custom web page rules through manual analysis are not applicable to a variety of complicated website page rule customization, which is time-consuming, labor-intensive, and low in efficiency

Images