Method for collecting internet data in one step

An Internet and data technology, applied in the field of information retrieval, can solve problems such as rising data collection costs, and achieve the effect of short collection time, avoiding data leakage, and omitting development.

Inactive Publication Date: 2021-09-17
浪潮卓数大数据产业发展有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

If the data collection program still extracts the full amount of target data according to the established display parameters of the website, the required collection time, resource nodes, processing and other data collection costs will inevitably increase significantly

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for collecting internet data in one step

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0026] The one-step method for collecting Internet data includes the following steps:

[0027] Step 1: Obtain request parameters;

[0028] Step 2: Customize the request parameters for website retrieval;

[0029] Step 3: Send an access request to the website through the user-defined request parameters, and obtain response information that matches the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention particularly relates to a method for collecting internet data in one step. The method for collecting internet data in one step comprises the following steps: firstly, collecting a request parameter, and carrying out custom setting on the request parameter for website retrieval; then sending an access request to the website through the user-defined request parameters, and obtaining the response information conforming to the user-defined data display parameters; carrying out one-time collection on website data to obtain total data; and finally, performing structured processing on the obtained retrieval result information, and storing the retrieval result information in a database to achieve the purpose of data collection. According to the method for collecting the internet data in one step, the request parameters related to data display are specified, so that the total target data of the website is accessed and collected at one time, the collection time is shorter, and collection resource nodes are not occupied, so that data collection omission is avoided, the data can be quickly and efficiently collected, development of a crawler program is simplified or omitted. The convenience is provided for subsequent data storage and data analysis.

Description

technical field [0001] The invention relates to the technical field of information retrieval, in particular to a method for collecting Internet data in one step. Background technique [0002] Modern society is a rapidly developing society. With the rapid development and popularization of computer and information technology, the scale of industrial application systems is expanding rapidly, and the data generated by industrial applications is growing explosively. People are increasingly aware of the importance of data. This concept has attracted widespread attention from practitioners and users in various industries. The mining and application of massive data heralds the arrival of a new wave of productivity growth and consumer surplus. With the development of government information disclosure and enterprise digitization, a large amount of valuable data can be obtained through the collection of Internet data. [0003] Information Retrieval (Information Retrieval), usually re...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F16/9538
CPCG06F16/9538G06F16/951
Inventor 邢荣宋设
Owner 浪潮卓数大数据产业发展有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap