Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and device for carrying out crawling task

A task and cloud server technology, applied in the Internet field, can solve problems such as low efficiency and IP addresses are easily blocked, and achieve the effect of improving crawling efficiency, avoiding blocking, and improving the efficiency of executing crawling requests

Inactive Publication Date: 2015-04-01
BEIJING GRIDSUM TECH CO LTD
View PDF6 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The main purpose of the present invention is to provide a method and device for performing crawling tasks to solve the problems in the prior art that using a single machine to perform crawling tasks is inefficient and the IP address is easily blocked

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for carrying out crawling task
  • Method and device for carrying out crawling task

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0021] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0022] It should be noted that the terms "fir...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a device for carrying out a crawling task. The method comprises the following steps: sending a received crawling request to a cloud distributed queue of a cloud server by a first terminal; reading the crawling request from the cloud distributed queue by a second terminal, wherein the second terminal is a cloud terminal; crawling a network resource according to the crawling request by the second terminal and storing crawling result data to a database; and reading the crawling result data from the database by the first terminal. Due to the adoption of the method and the device, the problems in the prior art that the efficiency of carrying out the crawling task by using a one machine is low and an IP address is easily shielded are solved, and the effect of improving the efficiency of carrying out the crawling task under the condition that the IP address is not shielded by a server is achieved.

Description

technical field [0001] The present invention relates to the field of the Internet, in particular to a method and device for performing crawling tasks. Background technique [0002] At present, the crawler program is a basic component of building an Internet search engine, and has the function of analyzing and crawling web page resources or other resources on the website. The crawler program can be divided into a link analysis module and a crawling module, wherein the crawling module is responsible for obtaining crawling results according to crawling requests. The realization of a common crawling module is to execute the crawling request locally, that is, the computer executing the crawler program directly sends a network request to the website or server to be crawled, and receives the server's response to the request. [0003] The existing technology mainly relies on the local computer to execute the crawling request, so the page crawling is completely dependent on the loca...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/951
Inventor 何恺铎
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com