Crawler implementation method, system and device and storage medium

A crawler and network card technology, applied in the field of network information, can solve problems such as waste of system resources, complex background management, and inability to guarantee real-time data, and achieve the effects of saving system resources, facilitating system maintenance and management, and reducing repetitive development work

Active Publication Date: 2019-05-28
携程旅游信息技术(上海)有限公司
View PDF6 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The data crawled by crawler technology usually needs to be used by the background data analysis system. Different background data analysis systems usually need to rewrite the related program code of crawler technology for different websites, so a lot of repeated development work is generated, and the background management is also very complicated.
[0003] At the same time, in order to maintain data security, the background data analysis system is usually placed in the intranet system. Therefore, in the process of crawling data and performing data analysis, it is necessary to first dump the data crawled by the crawler technology into the intranet and then be used Used by the background data analysis system, this method cannot guarantee the real-time performance of the data, and also wastes system resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Crawler implementation method, system and device and storage medium
  • Crawler implementation method, system and device and storage medium
  • Crawler implementation method, system and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals denote the same or similar structures in the drawings, and thus their repeated descriptions will be omitted.

[0026] Crawler technology is a method to collect information on the target website, and to open links and obtain information under the current link by looping through programming. The data crawled by crawler technology usually needs to be used by the background data analysis system. However, the existing technology usually does not take the background data analysis system into consideration as a whole. Therefore, it is usu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a crawler implementation method, system and device and a storage medium. The crawler implementation method comprises the steps of running a packaged crawler module in a host environment and opening an interface to a calling end; receiving a calling request from a calling end through the intranet network card, wherein the calling request comprises an address of a target website and a crawling mode; generating an executable script to execute the executable script according to the calling request, and crawling data from the target website through an external network card; feeding back the crawling data to a calling end through an intranet network card, so that different background data analysis systems share a crawler module in the mode, the repeated development is reduced, the data real-time performance is improved, and the system resources are saved.

Description

technical field [0001] The present invention relates to the technical field of network information, in particular, to a crawler implementation method, system, equipment and storage medium. Background technique [0002] With the blowout of the amount of data brought by the Internet, how to obtain data effectively and in real time has become an important issue in the Internet environment. Crawler technology is an important tool for network information acquisition. The data crawled by crawler technology usually needs to be used by the background data analysis system. Different background data analysis systems usually need to rewrite the related program code of crawler technology for different websites, so a lot of repeated development work is generated, and the background management is also very complicated. . [0003] At the same time, in order to maintain data security, the background data analysis system is usually placed in the intranet system. Therefore, in the process o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951H04L29/08
Inventor 宋海伟
Owner 携程旅游信息技术(上海)有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap