A data processing method and data processing device applied to a data warehouse

A data processing device and data warehouse technology, applied in the computer field, can solve the problems of source database pressure, data staleness, and insufficient timeliness, and achieve the effect of easy scheduling interval and simple update scheduling

Active Publication Date: 2019-02-05
BEIJING JINGDONG 360 DEGREE E COMMERCE CO LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] According to the above-mentioned method of offline batch extraction of data from the source database in the prior art, since the offline batch extraction can only use the SQL method to extract data through the database query engine, it will cause certain pressure on the source database
In order to reduce the pressure on the source database, data extraction from the source database is generally performed every night when the production pressure is low, resulting in a data delay of at least one day, and can only be updated by extracting data greater than each interval
In this way, the data obtained from the data warehouse query is relatively old, and the timeliness is not enough

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing method and data processing device applied to a data warehouse
  • A data processing method and data processing device applied to a data warehouse
  • A data processing method and data processing device applied to a data warehouse

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] The following describes exemplary embodiments of the present invention with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and should be regarded as merely exemplary. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Likewise, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In the following description, a mirror database of an online relational database (hereinafter referred to as the "first database") in a production environment stored in a data warehouse is taken as an example to illustrate the technical solution of this embodiment. The technical solution of the embodiment of the present invention will be described in detail. And...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method and data processing device applied to a data warehouse. The data processing method includes: when a first data base is operated, rectifying records of a mirror database in the first database in the data warehouse according to an incremental log corresponding to the operation and then storing the rectified records in a key / value database; fetching latest entries from the key / value database, additionally storing the entries in a first data table of the data warehouse and enabling the first data table to include historical versions of the records of the mirror database; searching data in the first data table. Therefore, timeliness of the data in the data warehouse can be improved.

Description

Technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method and a data processing device applied to a data warehouse. Background technique [0002] Data warehouse, the English name is Data Warehouse, which can be abbreviated as DW or DWH. A data warehouse is a strategic collection of all types of data that supports the decision-making process at all levels of the enterprise. The data comes from various scattered source databases, such as relational databases in a production environment, and databases where other data that needs to be analyzed are located. The data warehouse is created for the purpose of analytical reporting and decision support, providing companies with the need for business intelligence to guide business process improvement and monitor time, cost, quality and control. Compared with the source database, the data warehouse is a summary of the former data, has the characteristics of large cap...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/28G06F16/21
CPCG06F16/219G06F16/2329G06F16/254G06F16/283G06F16/284
Inventor 刘羽刘彦伟
Owner BEIJING JINGDONG 360 DEGREE E COMMERCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap