Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data construction and management method of data warehouse and terminal

A technology of data warehouse and management method, applied in the direction of database index, database model, database design/maintenance, etc., can solve the problems of complex structure of data warehouse, unstandardized data, and risk of small and medium-sized enterprises, so as to improve data extraction. Efficiency, reduce data governance costs, and reduce the effect of data redundancy

Pending Publication Date: 2022-04-12
XIAMEN MEIYA PICO INFORMATION
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Due to the lack of unified data planning, credible data sources and data standards, the data cannot be standardized and normalized, so there is a certain amount of redundancy
In addition, there is a problem of human intervention in the data processing process, and the establishment of a data warehouse in the presence of a large amount of messy data will cause unpredictable risks compared to small and medium-sized enterprises.
[0004] Therefore, in the current data governance, there are problems such as scattered business knowledge system, complex data warehouse structure, and many data extraction procedures, which lead to increased research and development costs of data warehouses and low data extraction efficiency, which cannot provide a fast and efficient data extraction for small and medium teams. , transform data, clean and encrypt data, store data in multiple dimensions

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data construction and management method of data warehouse and terminal
  • Data construction and management method of data warehouse and terminal
  • Data construction and management method of data warehouse and terminal

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0060] Please refer to figure 1 , a data construction and management method for a data warehouse, comprising:

[0061] S1. Obtain the original data table to be stored;

[0062] S2. Judging whether there is a standard table corresponding to the original data table to be stored in the data warehouse; specifically: the standard table includes a view table and a physical table; judging the standard table corresponding to the original data table to be stored Type, if it is a physical table, it is judged whether there is an input standard in the original data table to be stored, if so, it means that there is a standard table corresponding to the original data report to be stored; if it is a view table, then judge the Whether there is a corresponding access configuration in the original data table to be stored, if so, it means that there is a standard table corresponding to the original data report to be stored;

[0063] If it does not exist, execute S3, create a standard table cor...

Embodiment 2

[0089] The difference between this embodiment and Embodiment 1 is that protocol data based on HTTP, REST, etc. are used as the data input layer;

[0090] Change step S321 to S321b:

[0091] S3211. Obtain the request header and various parameters of the request header based on the kettle configuration;

[0092] S3212. Configure the splicing interface request body content;

[0093] S3213, configure the splicing interface request body, and output all parameter content of the request header to the log for post-test verification;

[0094] S3214. Configure the HTTP access component, configure the url address, the request protocol POST\GET protocol, and configure the domain name returned by the obtained result;

[0095] S3215. Write a java script to convert the returned data format into a common json format, and stipulate that the converted format is as follows:

[0096] {"page":"1","pageSize":"100","data":[{"keyField":"",...}]}

[0097] Among them, "page": "1" means: the current...

Embodiment 3

[0104] The difference between this embodiment and Embodiment 1 or 2 is that the data input layer is based on data such as file storage, columnar storage, relational database, memory database, full-text search, and document database;

[0105] Change step S3 to S3a, including steps:

[0106] S31: Execute the sql script to display the corresponding field header and type;

[0107] S32: Write table creation statements based on java script, generate standard creation statements, and generate standard md5 fields and timestamp fields; perform partitioning strategies based on the standard custom md5 fields and timestamp fields;

[0108] S33: create a table script: call the doris data source to execute the create statement;

[0109] Changing step S4 to S4b includes steps:

[0110] S41: Query the total number of data, and obtain the total number for the next step to obtain data by paging;

[0111] S42: Set query parameters: set the total number of queries to a constant, set the number...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data construction and management method for a data warehouse and a terminal, and the method comprises the steps: generating corresponding standard data from a to-be-stored original data table after the to-be-stored original data table is obtained, and then mapping the standard data with a standard field table to obtain a standard table, or directly creating the corresponding standard data based on the to-be-stored original data table, therefore, the original data table of the database and the generated standard data do not need to be stored in the database, data redundancy in the database is reduced, and rapid management and storage of the data are realized; meanwhile, the field data in the original data table is subjected to filtering, duplicate removal, format conversion, UDF function processing and the like, so that the redundancy of the stored data is reduced while the stored data is highly standardized, and therefore, the data extraction efficiency can be improved, and the data treatment cost can be reduced.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method for building and managing data in a data warehouse and a terminal. Background technique [0002] With the rapid development and wide application of computer network and database technology, information management in all walks of life has entered a new era. In the context of the current increasing amount of data, the government and enterprises have an urgent need for data services and data governance. Among them, in the process of asset data governance, there are various problems such as asset data confusion, data disorder, and data incompleteness, which make the data governance process complicated. Moreover, it is difficult for data governance to form standardized pipeline processing, which makes the governance effect low, and the quality of asset data after governance is poor. At the same time, after the governance process is programmed, it is still necessary to check t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/28G06F16/22G06F16/21
Inventor 刘远祥王仁斌张峰林镇荣
Owner XIAMEN MEIYA PICO INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com