MQTT-based real-time streaming data analysis processing method

A data analysis and processing, real-time streaming technology, applied in the direction of electrical digital data processing, resource allocation, inter-program communication, etc., can solve the problem of storage processing without any specifications, application restrictions, etc., and achieve the effect of good scalability

Pending Publication Date: 2021-07-23
上海寰果信息科技有限公司
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The MQTT protocol itself only regulates the subscription and publishing routing of the message itself, and there is no specification for the storage and processing of subsequent messages, which limits its application.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • MQTT-based real-time streaming data analysis processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] The specific implementation of the MQTT-based real-time stream data analysis and processing method of the present invention will be described in detail below.

[0025] See attached figure 1 , the present invention applies the streaming processing architecture of MQTT Borker + Broker Kafka Bridge + Kafka + KafkaStream, the Internet of Things device is used as the source of data, MQTT Broker is used as the data collector, and the collected data is published to Kafka in a centralized manner, and finally through the distributed Kafka Stream streams and processes data from IoT devices in real time.

[0026] The Kafka Connect MQTT connector is a plugin for sending and receiving data from an MQTT broker. MQTT brokers are persistent and provide MQTT-specific functionality. It consumes pushed data from IoT devices, which Kafka Connect pulls at its own pace without overwhelming or overwhelming the source. Out-of-the-box scalability and integration features such as Kafka Connec...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of communication data processing, and discloses an MQTT-based real-time streaming data analysis processing method, which comprises the following steps: (1) receiving user messages in a high-concurrency scene by using a Kafka partition mechanism, writing the user messages into a disk in a sequential writing manner, and storing the user messages in a message queue based on a publishing / subscribing mode; (2) searching and filtering the message through Kafka Stream, persistently storing the message into a database, and keeping monitoring on the state of a receiving end; sending the filtered data to an MQTT server, and storing the filtered data in different themes; and (3) enabling the receiving ends subscribing to different themes in the MQTT protocol to carry out matching to obtain messages, and enabling the receiving ends to send state messages to update the online list every time the receiving ends are online or offline. According to the invention, the data acquisition of the Internet of Things equipment and the control of the Internet of Things equipment are realized, the weight is light, the realization is simple, and the message pushing of the message Qos, TLS and the like is supported.

Description

technical field [0001] The invention relates to the technical field of communication data processing, in particular to an MQTT-based real-time stream data analysis and processing method. Background technique [0002] MQTT is an IoT transport protocol designed for lightweight publish / subscribe message transmission, aiming to provide reliable network services for IoT devices in low-bandwidth and unstable network environments. MQTT is a lightweight transport protocol developed specifically for the Internet of Things. The MQTT protocol is specially optimized for devices with low bandwidth networks and low computing power, making it adaptable to various IoT application scenarios. [0003] The MQTT protocol itself only regulates functions such as subscription and publishing routing of the message itself, and there is no specification for the storage and processing of subsequent messages, which limits its application. Contents of the invention [0004] The purpose of the presen...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F9/54
CPCG06F9/5083G06F9/546G06F2209/547G06F2209/548
Inventor 袁纯
Owner 上海寰果信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap