System and method for automatically associating real-time streaming data with service dimension information in cloud environment

Abstract

The invention provides an automatic association system and method for real-time streaming data and service dimension information in a cloud environment. The system comprises a metadata modeling module, a real-time data access module, a flow table association calculation module, a dimension data updating and storage module and a data persistence storage module. The metadata modeling module is used for carrying out abstract modeling on resources in the cloud computing platform to obtain model metadata; the real-time data access module is used for realizing real-time data access to the flow table association calculation module based on the model metadata; the flow table association calculation module is used for realizing automatic and efficient association of real-time flow data and dimension data; the dimension data updating and storing module is used for automatically updating dimension table data in the distributed cache; and the data persistence storage module is used for carrying out persistence storage on the real-time detailed data. According to the invention, efficient and automatic association processing operation of various original real-time data and business dimension data information generated by heterogeneous applications in a cloud environment is realized.

Description

technical field [0001] The invention relates to the field of cloud computing, in particular, to a system and method for automatically associating real-time stream data and business dimension information in a cloud environment. Background technique [0002] With the rapid development of cloud computing and big data-related businesses, in the face of massive, fast, and multi-dimensional real-time streaming data, the traditional offline data analysis method in the T+1 mode can no longer meet the business requirements for data timeliness. Although real-time stream processing technology can achieve quasi-real-time or even real-time requirements, real-time stream data often only contains a small amount of raw data, and it is impossible to directly perform multi-dimensional analysis on such data. Information association can be used by business systems such as large-screen display and analysis reports. [0003] The existing real-time data and business dimension information associat...

AI Technical Summary

Problems solved by technology

[0004] 1. The API-based implementation method requires users to code by themselves, which has a high degree of freedom, but the technical threshold is high, the implementation is difficult, the level of developers is uneven, the performance is difficult to guarantee, and the online deployment process is complicated;

[0005] 2. Compared with the API programming method, the SQL-based implementation method has a lower technical threshold to a certain extent, but users still need to complete the SQL writing by themselves. There are still problems with uneven SQL writing levels and difficult performance guarantees, but the difficulty is reduced. At the same time, it also brings a decrease in the degree of freedom, and it is impossible to accurately control the association process between real-time streaming data and business dimension information. For example, when the business dimension information changes, the granularity and timing of updates cannot be precisely controlled.

Images