A method for preventing website content from being crawled

Abstract

The invention provides a method for preventing website content from grabbing. Firstly, a rule for judging grabbing behavior is established; the WEB server side obtains client information, and then passes it to the anti-grabbing system server; Perform verification, and return the verification identification result to the WEB server side, and the WEB server side decides whether to execute the data query of the requested page or output a prompt of denial of access according to the verification result. The anti-grabbing method of the website content proposed by the present invention, through the strict formulation of the verification process, prevents from verifying the request of the client, effectively preventing the website data from being grabbed, and at the same time, in addition to the verification process, there is also a regular automatic update mechanism , to ensure the real-time update of the blacklist table and customer status table data, and to maintain the operation of the entire process more effectively and stably.

Description

technical field [0001] The invention relates to a method for preventing website content from grabbing. Background technique [0002] "Crawling" mentioned in this article refers to a way for programs to obtain data from other websites in accordance with specified rules. [0003] In the early years, a search engine system appeared on the Internet, a platform formed by crawling website content to achieve massive data. This technology obtains the website address through various channels, and crawls the content of the webpage according to the URL. The obtained content is analyzed and finally the corresponding data information is obtained; at the same time, there are also other non-search engine platforms for data capture. Competitors or other related companies can bring business value to them by capturing specific information content. [0004] The other kind of crawling is malicious. Competitors exist regardless of corporate websites or personal websites. In order to paralyze co...

AI Technical Summary

Problems solved by technology

[0005] According to the above mentioned search engine capture, business value capture, or malicious attack capture, there are two main problems: one is that large-scale data theft will bring certain business impacts to website operations, and there may be some Exposure of private data will have a negative impact on individuals or companies; second, whether it is normal crawling or malicious attack crawling, it will indirectly or directly affect the performance of the website server, thereby reducing the stability of the website, especially malicious ones. Attacks and crawling directly damage the interests of websites and enterprises

For the crawled websites, especially those with original content, these operations occupy a large amount of network resources of the crawled websites and reduce the speed and efficiency of the network; on the other hand, they also infringe Intellectual property rights of the crawled website, thus harming the interests of the crawled website

Images