Data collection method and device

A data and data collection technology, applied in the Internet field, can solve the problems of large data volume, low data quality, and many useless data, so as to achieve the effect of fast and effective collection and solve the problem of excessive data volume

Active Publication Date: 2017-10-24
ZTE CORP
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a method and device for collecting data, so as to at least solve the problems of excessive data volume, many useless data, and low data quality in the collection of water conservancy public opinion data in the related art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data collection method and device
  • Data collection method and device
  • Data collection method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] Hereinafter, the present invention will be described in detail with reference to the drawings and examples. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0025] It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence.

[0026] In this embodiment, a data collection method is provided, figure 1 is a flowchart of a data collection method according to an embodiment of the present invention, such as figure 1 As shown, the process includes the following steps:

[0027] Step S102: establishing a corresponding relationship between keywords used to search for data to be collected and corresponding webpage addresses containing keywords;

[0028] Step S104: recursively obtain...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data collection method and a data collection device. The method comprises the steps of building a corresponding relation between keywords used for searching data to be collected and corresponding webpage addresses including the keywords; circularly acquiring the webpage address in the corresponding relation; sending the acquired webpage address into a thread for crawling, and storing the crawled webpage content into a memory; acquiring text content of the webpage content in the memory in a preset manner, and storing the acquired text content into a file with a specified path. According to the method and the device provided by the invention, the problems that in related technologies for collecting water conservancy public opinion data, data size is large and too much data is invalid and data quality is low are solved.

Description

technical field [0001] The present invention relates to the field of the Internet, in particular, to a method and device for collecting data. Background technique [0002] With the rapid development of big data and new media, information dissemination has become faster and more convenient, which has led to uncertainty and uncontrollability of information management. Water pollution, drinking water safety, floods and droughts and other information are disseminated in a timely and rapid manner in the new media era, which not only monitors our water environment, but also provides an important channel for us to discover water environment problems in time and avoid potential water risks. Monitoring and responding to public opinion on water conservancy hotspots has become an important part of current water conservancy work. To this end, the Ministry of Water Resources has made deployments specifically for public opinion on water conservancy hotspots: establishing a special agency...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/951G06F16/00
Inventor 彭建华
Owner ZTE CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap