Supercharge Your Innovation With Domain-Expert AI Agents!

Big data storage center-oriented internet data acquisition system and acquisition method

A technology of data acquisition system and big data storage, applied in network data retrieval, network data indexing, electronic digital data processing and other directions, can solve the problem of not obtaining Internet data, and achieve the effect of facilitating data mining and analysis and enriching data

Inactive Publication Date: 2016-07-13
JIANGSU R & D CENTER FOR INTERNET OF THINGS
View PDF3 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, when many companies build big data storage centers, they only use data warehouses or middleware to collect and store the data of each subsystem, and do not obtain Internet data.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data storage center-oriented internet data acquisition system and acquisition method
  • Big data storage center-oriented internet data acquisition system and acquisition method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0036] Internet data acquisition systems for big data storage centers, such as figure 1 As shown, including data crawling server, data gateway, and database server; the data crawling server is connected to the data gateway, and the data gateway is connected to the database server;

[0037] On the data crawling server, a data crawling main program and a format processing program are established; the name of the data crawling main program established in the present invention is crawler, and users can input relevant parameters here to start executing tasks. The valid data crawled by the data crawling server from the third-party website through the crawler program is all retrieved from the other party's own database, so all data returned are in JSON format (JSON format is a common data packaging format for Web services) , at this time, it needs to be processed and...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a big data storage center-oriented internet data acquisition system. The system comprises a data crawling server, a data gateway and a database server, wherein the data crawling server is connected with the data gateway; the data gateway is connected with the database server; in the data crawling server, a main data crawling program and a format processing program are established; in the data crawling server, at least one target folder is further established and each target folder corresponds to a target website; in each target folder, a crawling program uniquely corresponding to each target website and a target address file are established, and the target address file stores a URL link of target content; and the data gateway as a transfer station is in charge of connecting the data crawling server with the database server and transferring website data information captured by the data crawling server to the database server. According to the system, the internet data can be effectively captured and a good foundation is laid for subsequent data analysis.

Description

technical field [0001] The invention relates to the technical field of data collection, in particular to an Internet data collection system. Background technique [0002] With the rapid development of Internet information technology, enterprise data capture, storage, analysis, processing and application have become very convenient, and enterprise strategic decision-making and crisis management are shifting towards data-driven prediction, development and decision-making. Therefore, future decision-making behaviors in response to competition and crises will be based on the capture and analysis of data, rather than the traditional mode of relying on experience and intuitive judgment. [0003] To establish a "big data strategy" system, the first thing to achieve is data capture, because the analysis, interpretation and application of various types of data must be carried out on the collected data. Only through the comprehensive and accurate collection of the required data, Form...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/9566G06F16/951
Inventor 王军军刘斌台宪青
Owner JIANGSU R & D CENTER FOR INTERNET OF THINGS
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com