Acquisition method and device of web crawler request

A technology of web crawler and acquisition method, which is applied in the field of acquisition of web crawler requests, can solve problems such as low efficiency of web crawling, and achieve the effect of improving user experience and efficiency

Active Publication Date: 2015-03-11
BEIJING GRIDSUM TECH CO LTD
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The main purpose of the present invention is to provide a method and device for obtaining a web crawler request, to solve the problem of low web crawling efficiency caused by manual intervention in the verification code login process in the prior art to a certain extent

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Acquisition method and device of web crawler request
  • Acquisition method and device of web crawler request
  • Acquisition method and device of web crawler request

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other if there is no conflict. Hereinafter, the present invention will be described in detail with reference to the drawings and in conjunction with the embodiments.

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "first" an...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an acquisition method of a web crawler request and an acquisition device thereof. The acquisition method comprises the steps: acquiring verification code images of to-be-crawled resource information; performing division and noise reduction to the verification code images to obtain multiple first images; performing binarization to all first images to obtain the first matrixes of all first images; reading the second matrixes corresponding to the first matrixes from a database; acquiring characters indicated by the second matrixes to obtain verification code information; generating the web crawler request by the verification code information and the user information in advance acquired, wherein the web crawler request is used to acquire the resource information. Thus, through the method and the device, the low web crawling efficiency caused by manual intervention is partly solved in the existing technology, and the web crawling efficiency effect is improved.

Description

Technical field [0001] The present invention relates to the Internet field, and in particular, to a method and device for obtaining a web crawler request. Background technique [0002] With the rapid development of the Internet, network resources have become more and more abundant, and the artificial search for such information is far from satisfying the demand. How to effectively extract and use this information has become a huge challenge. In order to solve the effective and intelligent extraction of information, web crawlers came into being. Before a web crawler obtains a specific resource, it first constructs the corresponding request header information, and sends a request to a specific site in the form of Post or Get cookie settings, and then obtains the response resource information. However, resource information of some sites on the Internet can only be obtained after logging in, and even some sites require verification code information to be entered when logging in for ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/951
Inventor 李庆
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap