Supercharge Your Innovation With Domain-Expert AI Agents!

Crawler processing method and device, server and computer readable storage medium

A processing method and crawler technology, applied in the computer field, can solve the problems of reducing crawler efficiency, lack of pertinence, and low scalability, and achieve the effect of improving crawler efficiency and scalability.

Pending Publication Date: 2020-02-28
PING AN TECH (SHENZHEN) CO LTD
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to the lack of pertinence and low scalability of this method, the crawler efficiency is reduced

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Crawler processing method and device, server and computer readable storage medium
  • Crawler processing method and device, server and computer readable storage medium
  • Crawler processing method and device, server and computer readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0055] see figure 1 , is a schematic flowchart of a crawler processing method provided in the embodiment of the present application. The method can be applied to a server, and the server can be a server or a server cluster in the Internet. Specifically, the method may include the following steps:

[0056] S101. Receive a task start instruction for a specified grabbing task sent by a terminal, the task start instruction includes a first configuration record, a second configuration record, and a third configuration record of the specified grabbing task, and the first configuration record Including seed information, the second configuration record includes crawler configuration information set for each type of page to be crawled in at least one type of page to be crawled, and the third configuration ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a crawler processing method and device, a server and a computer readable storage medium, and the method comprises the steps: receiving a task starting instruction, transmitted by a terminal, of a specified grabbing task, and the task starting instruction comprises a first configuration record, a second configuration record and a third configuration record of the specified grabbing task; according to the seed information included in the first configuration record and the crawler configuration information included in the second configuration record for indicating each type of to-be-grabbed page in the at least one type of to-be-grabbed pages, executing crawler operation to obtain a crawler data set corresponding to each type of to-be-grabbed page; andaccording to an analysis rule corresponding to each type of to-be-grabbed page included in the third configuration record, analyzing target data from each page included in the crawler data set corresponding to each type of to-be-grabbed page. By adopting the method and the device, the crawler process can be more targeted, the expandability can be improved, and the crawler efficiency can be improved.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a crawler processing method, device, server and computer-readable storage medium. Background technique [0002] With the development of network technology, the network contains more and more data. If you want to obtain data, you can usually use crawler technology to obtain data from web pages or databases. [0003] As an important means of obtaining network data, crawlers are closely related to the difficulty of obtaining network data and the data source website. Traditional crawling systems need to write specific codes for different data sources to crawl. During the crawling process, the crawling task is generally performed based on a large amount of data provided by a link address. However, due to the lack of pertinence and low scalability of this method, the crawler efficiency is reduced. Contents of the invention [0004] The embodiments of the present ap...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951
CPCG06F16/951
Inventor 杜晓宇
Owner PING AN TECH (SHENZHEN) CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com