Data processing method, crawler device, medium and electronic equipment

A data processing and crawler technology, applied in the field of computer networks, can solve the problem of crawling with few APPs

Pending Publication Date: 2021-07-30
北京精准沟通传媒科技股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] Existing crawler tools include, for example, Octopus, which pre-crawls the data of mainstream websites. Users can select certain types of data on the website they need in this tool, such as title, price, name, etc. This tool It is mainly designed for novice users. The advantage is that it is easy to use, but the disadvantage is that it can only meet simple needs; in addition, there is

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, crawler device, medium and electronic equipment
  • Data processing method, crawler device, medium and electronic equipment
  • Data processing method, crawler device, medium and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] Various aspects and features of the present disclosure are described herein with reference to the accompanying drawings. These and other characteristics of the invention will become apparent from the following description of preferred forms of embodiment given as non-limiting examples with reference to the accompanying drawings.

[0017] This specification may use the phrases "in one embodiment," "in some embodiments," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may refer to one or more of the same or type of embodiments in accordance with the present disclosure. Note that throughout the specification, the same or similar reference numerals designate the same or similar elements, and unnecessary repeated descriptions are omitted. In addition, in a specific embodiment, an element appearing in a singular form does not exclude that it may appear in a plural (plural) form.

[0018] Such as figure 1 As shown, a data processing me...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data processing method, a crawler device, a medium and electronic equipment. The data processing method comprises the following steps: acquiring configuration data related to a crawler task; obtaining return data downloaded by executing the crawler task; analyzing the return data by using information which is obtained from the configuration data and is related to analysis of the return data to obtain result data expected to be extracted; determining whether the type of the returned data is a first type, if not, converting the result data into the result data of the first type, otherwise, not converting the result data; and storing the result data of the first type. According to the scheme of the invention, the crawled data can be managed uniformly without being limited by the type of the crawled target object, and subsequent management, maintenance and various processing of the crawled data are facilitated.

Description

technical field [0001] The invention relates to the technical field of computer networks, in particular to a data processing method, a crawler device, a medium and electronic equipment. Background technique [0002] Existing crawler tools include, for example, Octopus, which pre-crawls the data of mainstream websites. Users can select certain types of data on the website they need in this tool, such as title, price, name, etc. This tool It is mainly designed for novice users. The advantage is that it is easy to use, but the disadvantage is that it can only meet simple needs; in addition, there is a crawler tool named Locomotive Collector, which is designed for users with relevant crawler experience. Some simple configurations are performed on the website, including submitting URLs, expressions, etc., but it also cannot meet more complex requirements. [0003] Existing crawler tools are generally developed for the PC side, crawling webpage data on the PC, and rarely crawling...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F16/955
CPCG06F16/951G06F16/955
Inventor 杨光周天星
Owner 北京精准沟通传媒科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap