Data processing method and data processing device applied to data warehouse

A data processing device and data warehouse technology, applied in the computer field, can solve the problems of source database pressure, outdated data, insufficient timeliness, etc., and achieve the effect of easy scheduling interval and simple update scheduling.

Active Publication Date: 2015-01-21
BEIJING JINGDONG 360 DEGREE E COMMERCE CO LTD
View PDF5 Cites 28 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] According to the above-mentioned method of offline batch extraction of data from the source database in the prior art, since the offline batch extraction can only use the SQL method to extract data through the database query engine, it will cause certain pressure on the source database
In order to reduce the pressure on the source database, data extraction from the source database is generally performed every night when the production pressure is low, resulting in a data delay of at least one day, and can only be updated by extracting data greater than each interval
In this way, the data obtained from the data warehouse query is relatively old, and the timeliness is not enough

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and data processing device applied to data warehouse
  • Data processing method and data processing device applied to data warehouse
  • Data processing method and data processing device applied to data warehouse

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0027] In the following description, the technical solution of this embodiment is described in detail by taking the mirror database of an online relational database (hereinafter referred to as "the first database") stored in the production environment by the data warehouse as an example. , and the technology of the Hadoop system is adopted in this embodiment. figure 1 is ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method and data processing device applied to a data warehouse. The data processing method includes: when a first data base is operated, rectifying records of a mirror database in the first database in the data warehouse according to an incremental log corresponding to the operation and then storing the rectified records in a key/value database; fetching latest entries from the key/value database, additionally storing the entries in a first data table of the data warehouse and enabling the first data table to include historical versions of the records of the mirror database; searching data in the first data table. Therefore, timeliness of the data in the data warehouse can be improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data processing method and a data processing device applied to a data warehouse. Background technique [0002] Data warehouse, the English name is Data Warehouse, which can be abbreviated as DW or DWH. A data warehouse is a strategic collection of all types of data that support decision-making processes at all levels of an enterprise. Its data comes from various scattered source databases, such as relational databases in the production environment, and other databases where the data that needs to be analyzed resides. Data warehouses are created for analytical reporting and decision support purposes, providing businesses with the business intelligence they need to guide business process improvement and monitor time, cost, quality and control. Compared with the source database, the data warehouse is a summary of the former data, which has the characteristics of large capacity...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/219G06F16/2329G06F16/254G06F16/283G06F16/284
Inventor 刘羽刘彦伟
Owner BEIJING JINGDONG 360 DEGREE E COMMERCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap