Stream computing immediate processing system based on Storm

Abstract

The invention discloses a stream computing immediate processing system based on Storm. The system comprises four parts, wherein a first part is that data is accessed into the part to obtain data froma front-end business system; the second part is a most important Storm real-time processing part, the data is accessed from an access layer, and the data is transmitted into a data landing layer afterreal-time processing; the third layer is the data landing layer, and the part appoints a data landing way; and the fourth part is a metadata manager. Through the investigation and survey of practicalbusiness requirements and algorithm research, a data processing rule base which meets the practical business is met, and a basic event is efficiently detected in real time according to an event modelpredefined by the user until all basic events which form an event model are detected. The system is suitable for processing complex analysis and calculation which has an extremely high instantaneityrequirement on a data stream which conforms to power grid business data features, and a purpose that data analysis speed is quickened and the real-time analysis and calculation is realized can be achieved.

Description

technical field [0001] The invention relates to the technical field of intelligent power distribution and utilization, in particular to a Storm-based stream computing real-time processing system. Background technique [0002] At present, many data management systems used in institutions and organizations at home and abroad have solved the problems of low latency and high concurrency, but such systems may not be fully suitable for the storage requirements of business systems in the power grid. The current research results lack reliable, scalable, and processing methods that can handle large-scale data and more complex requirements, especially in the unstructured data processing and real-time stream computing. There is still a lack of effective typical applications in the power industry, and further research is needed. Research flow computing technology and data query technology that meet the characteristics of the power industry. [0003] In terms of stream computing, the ex...

AI Technical Summary

Problems solved by technology

The existing limitations include: first, the existing solutions use limited hardware resources to process large-scale data in a relatively short period of time, and need to be deeply integrated with intensive distributed frameworks such as MapReduce; second, the existing distributed data The main purpose of stream technology is to reduce network overhead, and network bandwidth is not the only resource bottleneck of the system

Images