Real-time data warehouse, real-time data processing method, electronic device and storage medium

A real-time data processing and real-time data technology, applied in the computer field, can solve problems such as low development efficiency and poor scalability

Active Publication Date: 2019-09-17
北京三快云计算有限公司
View PDF11 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] This application provides a real-time data warehouse to solve the problems of poor scalability and low development efficiency of real-time data warehouses in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time data warehouse, real-time data processing method, electronic device and storage medium
  • Real-time data warehouse, real-time data processing method, electronic device and storage medium
  • Real-time data warehouse, real-time data processing method, electronic device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] A real-time data warehouse disclosed in this embodiment, such as figure 1 As shown, the data warehouse includes at least the following three layers arranged sequentially from bottom to top:

[0026] Source data layer 110, used for real-time acquisition and storage of business system data;

[0027] A multi-dimensional detail layer 120, configured to process the data in the source data layer 110 based on preset dimensions, and store the processed data;

[0028] The application layer 130 is configured to process the data stored in the multi-dimensional detailed layer 120 in response to the database operation, and output the processed data;

[0029] Among them, real-time data processing technology is used to process data in each layer, and data transmission is performed between layers through message queue and / or cache database.

[0030] In specific implementation, such as figure 2 As shown, the source data layer 110 includes a message queue 1101 . The source data laye...

Embodiment 2

[0057] Based on Embodiment 1, a real-time data warehouse disclosed in this embodiment, such as image 3 As shown, between the multi-dimensional detail layer 120 and the application layer 130, it also includes:

[0058] The summary layer 140 is configured to perform summary processing on the data stored in the multi-dimensional detail layer 120 based on at least one preset dimension.

[0059] The construction principle of the summary layer 140 is to find some dimensions that are often analyzed or used in data applications, such as users, merchants, etc., and for the data stored in the multi-dimensional detail layer 120, according to the above-mentioned commonly used dimensions, such as users, merchants, etc. Merchants, etc. are aggregated.

[0060] The summary layer 140 also includes at least one real-time data processing module, such as image 3 1401 to 1403 in. The at least one real-time data processing module is used for processing the data of the corresponding subject. ...

Embodiment 3

[0065] Correspondingly, the embodiment of the present application also discloses a real-time data processing method, which is applied to such as figure 1 As shown in the real-time data warehouse, the real-time data warehouse at least includes the following source data layer, multi-dimensional detailed layer and application layer arranged in sequence from bottom to top.

[0066] During specific implementation, the real-time data warehouse is divided into multiple layers according to the key nodes in the data processing flow. In specific implementation, combined with the demand for real-time data processing speed, the real-time data warehouse can be divided into three layers, namely: source data layer, multi-dimensional detailed layer and application layer.

[0067] The source data layer is used to obtain and store data of the business system in real time; the multi-dimensional detailed layer is used to process the data of the source data layer based on preset dimensions and st...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a real-time data warehouse, belongs to the technical field of computers, and solves the problems of poor expandability and low development efficiency of the real-time data warehouse in the prior art. The real-time data warehouse at least comprises the following three layers arranged in sequence from bottom to top: a source data layer used for obtaining and storing data of abusiness system in real time, a multidimensional detail layer used for processing the data of the source data layer based on preset dimensions and storing the processed data, and an application layerused for processing the data stored in the multidimensional detail layer in response to database operation and outputting the processed data, wherein the data is processed through a real-time data processing technology in the layers; and the layers perform data transmission through a message queue and / or a cache database. The data processing processes are layered in combination with business characteristics according to the data processing processes, so that the data processing complexity is lowered and the data warehouse development efficiency is improved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a real-time data warehouse, a real-time data processing method, electronic equipment and a storage medium. Background technique [0002] With the development of Internet technology, there are more and more network applications, and the network data generated in real time is gradually increasing. The demand for real-time mining of Internet data is gradually increasing. More and more business systems, based on real-time data mining technology, provide users with more Excellent service. Therefore, the demand for real-time data warehouse construction is increasingly urgent. In the prior art, when a business system builds a real-time data warehouse, it usually adopts an object-oriented development mode, and establishes a set of data processing procedures independently based on each type of business data. Therefore, every time a business type is added, the processing flow...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2455G06F16/28
Inventor 杨德臣
Owner 北京三快云计算有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap