Internet of Things big data analysis method based on stream computing and novel column type storage

A columnar storage and analysis method technology, which is applied in the field of big data analysis of the Internet of Things based on flow computing and new columnar storage, can solve the problems of large limitations, inability to update a single piece of data, and low throughput, so as to achieve convenient and rapid implementation Achieved effect

Inactive Publication Date: 2019-06-28
成都古河云科技有限公司
View PDF3 Cites 28 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is to provide a big data analysis method for the Internet of Things based on stream computing and new columnar storage, which solves the problem that the existing large data storage and analysis method for massive device information in the Internet of Things field cannot analyze batch data. Simultaneously update a single piece of data, and the problem of low throughput and large limitations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Internet of Things big data analysis method based on stream computing and novel column type storage
  • Internet of Things big data analysis method based on stream computing and novel column type storage
  • Internet of Things big data analysis method based on stream computing and novel column type storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] Explanation of technical terms:

[0053] Hadoop is a distributed system infrastructure developed by the Apache Foundation;

[0054] HDFS, Hadoop Distributed File System (Hadoop Distributed File System), has high fault tolerance and is designed to be deployed on low-cost (low-cost) hardware; and it provides high throughput (highthroughput) to access applications Program data, suitable for applications with large data sets.

[0055] Hive, a data warehouse tool based on Hadoop, can map a structured data file into a database table, and provide a simple SQL query function, which can convert SQL statements into Map Reduce tasks for execution.

[0056] HBase, the full name of Hadoop Database, is developed and implemented based on the idea of ​​Big Table. It provides a distributed data storage engine in a big data scenario, with linear scalability, strict consistency model, automatic fragmentation strategy, and powerful failure Resilience, good integration with Map Reduce, an...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an Internet of Things big data analysis method based on stream computing and novel column storage, and belongs to the technical field of Internet of Things big data analysis. The method comprises the following steps: firstly, carrying out communication protocol conversion on acquired data to obtain a unified data format, transmitting the unified data format into a message queue, and then transmitting the unified data format into a distributed stream computing framework Flink; then, the flow calculation framework Flink carries out data statistics, data aggregation and analysis on the data and then writes the data into Kudu in real time; calling an original interface (Scan) of the Kudu to carry out batch offline analysis on the data stored in the Kudu; by means of thebig data storage and analysis method based on the Internet of Things, the problems that according to an existing big data storage and analysis method of mass equipment information in the field of Internet of Things, single data updating cannot be conducted while batch data are analyzed, the throughput is small, and limitation is large are solved. The big data storage and analysis method based onthe Internet of Things solves the problem that the existing big data storage and analysis method of mass equipment information is high.

Description

technical field [0001] The invention belongs to the technical field of big data analysis of the Internet of Things, and relates to a big data analysis method of the Internet of Things based on stream computing and novel columnar storage. Background technique [0002] In the new era, the new generation of information and communication technology represented by the Internet of Things, big data, cloud computing, and artificial intelligence is developing rapidly. Through the continuous integration of the Internet of Things technology and other new technologies, the application of the Internet of Things is also showing an explosive trend, further realizing Information sharing between people, machines, and things is achieved, and a closer information flow ecosystem is formed between the information world and the physical world. In the Internet of Things, all things are interconnected, and the data has typical characteristics such as heterogeneity, massiveness, and real-time. In or...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/06H04L29/08G06F16/2458G06F16/28
Inventor 彭安
Owner 成都古河云科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap