Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A distributed real-time processing method for multi-source stream data in rail transit

A distributed real-time, distributed processing technology, applied in the field of data processing, to achieve high performance

Active Publication Date: 2021-08-17
ZHEJIANG BANGSUN TECH CO LTD +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The present invention proposes a distributed real-time processing method for rail transit multi-source stream data, aiming to provide certain real-time processing support for the big data platform of the rail transit system. Aiming at the data characteristics of the rail transit system, the present invention at least mainly solves the following Three technical questions:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A distributed real-time processing method for multi-source stream data in rail transit
  • A distributed real-time processing method for multi-source stream data in rail transit
  • A distributed real-time processing method for multi-source stream data in rail transit

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] The present invention will be further described below in conjunction with drawings and embodiments.

[0054] The present invention is used for the distributed real-time processing of multi-source stream data in rail transit systems. According to the characteristics of big data in rail transit systems, it can be mainly divided into the following two parts: the merging of multi-source stream data and the distributed processing of merged stream data. deal with.

[0055] Due to the complexity of the rail transit system, the collected data are usually different data streams including multiple different dimensions, so its merging can be divided into the following two steps:

[0056] Step 1: Dimensionally merge the real-time data of the same vehicle on the same track line. For example figure 1 The simplified model is shown.

[0057] The three data streams A, B, and C of the same vehicle in different dimensions are as follows: A stream data contains the vehicle’s unique iden...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed real-time processing method for multi-source flow data in rail transit. The method includes two parts: merging of multi-source flow data and distributed processing of merged flow data; The real-time data of the same vehicle on the same vehicle is merged in dimension, and the new flow obtained after dimension merger is merged in breadth; the distributed processing of the merged flow data is realized on the distributed system, and the distributed system has two types of Manager , which are JobManager and TaskManager respectively; multiple JobManagers are set; the present invention has a certain degree of scalability, and the scaling of the entire architecture will not reduce or increase the overall flow processing calculation; the present invention has the characteristics of high performance; the present invention is distributing In the process of processing, the distributed multi-JobManager state synchronization method is adopted to realize the complete distributed processing.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a distributed real-time processing method for multi-source flow data of rail transit. Background technique [0002] In view of the big data processing scenarios of the new rail transit system and its distributed storage and real-time processing requirements, traditional databases and data clusters are often used to meet the changing business needs when processing and querying mixed temporal big data in the full life cycle of the rail transit system. Multiple data entities need to be associated. The traditional SQL query is mainly applicable to large-scale batch processing, and the performance is not good when performing real-time stream processing. [0003] In stream processing scenarios, Spark and Flink are currently two of the more popular stream processing engines. They provide two operations, ConnectedStream and union, for data merging. The former only sup...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2455
CPCG06F16/24556G06F16/2456G06F16/24568
Inventor 高杨王刚黄滔鲍迪恩
Owner ZHEJIANG BANGSUN TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com