Method and device for acquiring external data by using crawler

A technology for external data and data acquisition, applied in the computer field, it can solve problems such as high labor consumption, no solution found, single function, etc., and achieve the effect of reducing manual operation, realizing automatic scheduling, and improving efficiency.

Pending Publication Date: 2019-08-30
PING AN TECH (SHENZHEN) CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In the existing technology, there are many technologies in the crawler field, but the function of each technology is too single, and the automation of the crawler and the data persistence of the crawler are relatively lacking, but after the crawler obtains the data, the user needs to further filter And processing, the efficiency is low, and it needs to consume a lot of manpower when it is applied to large-scale database establishment and periodic tasks
[0006] For the above-mentioned problems existing in related technologies, no effective solution has been found yet

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for acquiring external data by using crawler
  • Method and device for acquiring external data by using crawler
  • Method and device for acquiring external data by using crawler

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0037] The method embodiment provided in Embodiment 1 of the present application may be executed in a mobile terminal, a server, a computer terminal or a similar computing device. Take running on the server as an example, figure 1 It is a block diagram of the hardware structure of a server that uses crawlers to obtain external data according to an embodiment of the present invention. Such as figure 1 As shown, the server 10 may include one or more ( figure 1 Only one is shown in ) processor 102 (processor 102 may include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA) and memory 104 for storing data. Optionally, the above-mentioned server can also be A transmission device 106 for communication functions and an input and output device 108 are included. Those of ordinary skill in the art can understand that, figure 1 The shown structure is only for illustration, and it does not limit the structure of the above server. For ...

Embodiment 2

[0062]In this embodiment, a device for obtaining external data by using a crawler is also provided, and the device is used to implement the above embodiments and preferred implementation modes, and what has been described will not be repeated. As used below, the term "module" may be a combination of software and / or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

[0063] Figure 5 is a structural block diagram of a device for obtaining external data using a crawler according to an embodiment of the present invention, such as Figure 5 As shown, the device includes:

[0064] An acquisition module 50, configured to acquire a data acquisition instruction according to a trigger condition;

[0065] Calling module 52, for calling crawler program according to described data ac...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a method and a device for acquiring external data by using a crawler. On one hand, the method comprises the following steps: acquiring a data acquisition instruction according to a trigger condition; calling a crawler program according to the data acquisition instruction; receiving a crawler page captured by the crawler program; and analyzing the crawler page to obtain result data, and storing the result data in a mysql database. Through the method and the device, the technical problem that a crawler program cannot be automatically called to obtain datain the prior art is solved, the efficiency of capturing the data by using a crawler is improved, and manual operation is reduced.

Description

[0001] 【Technical field】 [0002] The invention relates to the field of computers, in particular to a method and device for acquiring external data by using a crawler. [0003] 【Background technique】 [0004] In the prior art, a crawler is a program or script that automatically captures information on the World Wide Web according to certain rules. A crawler is currently the most common and important means for all companies to obtain external data, and it can play a very good role in business data. supplementary effect. [0005] In the existing technology, there are many technologies in the crawler field, but the function of each technology is too single, and the automation of the crawler and the data persistence of the crawler are relatively lacking, but after the crawler obtains the data, the user needs to further filter And processing, the efficiency is low, and it needs to consume a lot of manpower when it is applied to large-scale database establishment and periodic tasks....

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F16/28
CPCG06F16/951G06F16/284
Inventor 申超波阮晓雯徐亮
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap