Supercharge Your Innovation With Domain-Expert AI Agents!

Method for building real-time counting warehouse frame and real-time counting warehouse frame

A framework and database technology, applied in the computer field to improve development efficiency, good support, and meet business needs

Pending Publication Date: 2022-03-08
SHENZHEN WORKEC TECH
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a method for building a real-time data warehouse framework, aiming to solve the problem of providing a real-time data warehouse framework that supports update operations, can meet business requirements, and can execute SQL-like statements

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for building real-time counting warehouse frame and real-time counting warehouse frame
  • Method for building real-time counting warehouse frame and real-time counting warehouse frame
  • Method for building real-time counting warehouse frame and real-time counting warehouse frame

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0028] In this embodiment, a method for building a real-time data warehouse framework is provided, such as figure 1 As shown, the method includes:

[0029] Step 101, saving the structured query language statement through the page configuration, so that the first engine reads the page configuration and executes the corresponding SQL statement;

[0030] Step 102, establishing a connection between the first engine and the second engine through a preset connector;

[0031] Step 103, configuring the parsing module, wherein the parsing module is used to parse the business messages in the first database into a custom format in real time, and write the parsed business messages into the stream processing platform;

[0032] Step 104, adding a formatter to the first engine, so that the first engine supports the custom format;

[0033] Step 105, configuring a unified data warehouse query framework to perform real-time statistics on the business messages through the unified data warehous...

Embodiment 2

[0041] In this embodiment, a business message processing method based on a real-time data warehouse framework is provided, such as image 3 As shown, the method includes:

[0042] Step 201, obtaining business messages in the first database, and parsing the business messages into a custom format;

[0043] Step 202, sending the parsed business message in the custom format to the stream processing platform;

[0044] Step 203, the first engine queries the business message in the stream processing platform by executing a structured query language statement, performs stream processing on the business message and writes it into the stream processing platform;

[0045] In step 204, the second engine collects the business messages in the stream processing platform into the unified data warehouse query framework for real-time statistics.

[0046] In this embodiment, the processing of business messages can be realized based on the real-time data warehouse framework. In a specific appli...

Embodiment 3

[0052] as image 3 For the specific implementation of the method, the embodiment of the present application provides a business message processing device based on the real-time data warehouse framework, such as Figure 4 As shown, the device includes:

[0053] A message acquisition module, configured to acquire business messages in the first database, and parse the business messages into a custom format;

[0054] A sending module, configured to send the parsed business message in the custom format to the stream processing platform;

[0055] A writing module, used for the first engine to query the business messages in the stream processing platform by executing a structured query language statement, and write the business messages into the stream processing platform after stream processing;

[0056] The processing module is used to use the second engine to collect the business messages in the stream processing platform into the unified data warehouse query framework for real-...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention is suitable for the technical field of computers, and provides a method for building a real-time data warehouse framework and the real-time data warehouse frame.The method comprises the steps that structured query language statements are stored through page configuration, and the page configuration is read through a first engine to execute the corresponding SQL statements; establishing connection between the first engine and a second engine through a preset connector; an analysis module is configured, and the analysis module is used for analyzing the service message in the first database into a self-defined format in real time and writing the analyzed service message into a stream processing platform; adding a formatting program to the first engine to enable the first engine to support the custom format; and configuring a uniform number warehouse query framework, so as to carry out real-time statistics on the service messages through the uniform number warehouse query framework. And the system development efficiency can be improved.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a method for building a real-time data warehouse framework and the real-time data warehouse framework. Background technique [0002] Phoenix is ​​a big data framework based on HBase, which can be well integrated with HBase, and supports operations such as update very well. Flink Sql is a new framework of Flink, which supports stream processing of data based on similar SQL statement formats, but Flink Sql currently does not support Phoenix. Canal can synchronize Mysql data to Kafka, but this component is limited by the data format requirements and cannot fully meet complex business needs. Flink Sql supports the Canal data format, but it is also limited by the component data format, making it difficult to support complex businesses need. [0003] For complex business, there is currently no real-time data warehouse framework that can support the following features at...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2458G06F16/27
CPCG06F16/2471G06F16/27
Inventor 张星亮黄育源
Owner SHENZHEN WORKEC TECH
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com