Big data platform data warehouse data processing method and device and computer equipment

A big data platform and data warehouse technology, applied in the field of devices, computer equipment, and data warehouse data processing methods of big data platform, can solve the problems such as the delay of batch data running at the data warehouse layer, and avoid repeated processing and delay. Low, reduce the effect of occupying system resources

Pending Publication Date: 2019-04-09
PING AN TECH (SHENZHEN) CO LTD
View PDF0 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The main purpose of this application is to provide a big data platform data warehouse data processing method, device and computer equipment, aiming to solve the problem of serious delay in data warehouse layer running batch processing data in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data platform data warehouse data processing method and device and computer equipment
  • Big data platform data warehouse data processing method and device and computer equipment
  • Big data platform data warehouse data processing method and device and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

[0037] refer to figure 1 , the embodiment of the present application provides a data processing method of a big data platform data warehouse, the data warehouse layer of the data warehouse includes a data preprocessing layer and a data summary layer, and the data summary layer depends on the data preprocessing layer.

[0038] The above data summary layer depends on the data preprocessing layer, that is, the relationship between the data preprocessing layer and the data summary layer is the upper and lower layers.

[0039] The above processing method includes the steps of:...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a big data platform data warehouse data processing method and device and computer equipment. A data warehouse layer of a data warehouse comprises a data preprocessing layer anda data summarizing layer, and the data summarizing layer depends on the data preprocessing layer; The processing method comprises the steps of obtaining a source data table of a source data layer, and preprocessing the source data table through a data preprocessing layer to obtain a public data table; And summarizing the public data table and other data tables which do not need to be preprocessedthrough the data summarizing layer. According to the invention, the data warehouse layer of the data warehouse is divided into the data preprocessing layer and the data summarizing layer; According to the method, the common data table is obtained by cleaning the data and placing the data in the data preprocessing layer, repeated processing on the same source data table is not needed, repeated processing of the data is avoided, occupation of system resources is reduced, the same data processing amount is achieved, and the delay condition of the method is far lower than that of the prior art.

Description

technical field [0001] The present application relates to the field of big data processing, in particular to a data processing method, device and computer equipment of a big data platform data warehouse. Background technique [0002] The traditional big data platform data warehouse is generally divided into source data layer (ODS), data warehouse layer (DW), data mart layer (DM), and data application layer (APP); among them, the data warehouse layer is the source data layer. Cleaning (deduplication, dirty data processing, etc.), and summarizing fields with the same granularity, so as to provide better use to the upper data mart layer. In actual projects, the data processing load of the data warehouse layer is the heaviest. On the one hand, it is necessary to clean dirty data and duplicate data, and on the other hand, it is necessary to associate multiple tables of the same granularity into one wide table; in the daily system During batch running, it often happens that the d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25
Inventor 许浩奇欧阳智张小松
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap