Data processing method and device, equipment and storage medium

A data processing and physical storage technology, applied in the computer field, can solve problems such as increasing the storage size, increasing the cardinality of physical storage columns, and reducing the compression ratio

Active Publication Date: 2019-11-29
北京博睿宏远数据科技股份有限公司
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Especially in the On-line Analytical Processing (OLAP) analysis of application performance management (Application Performance Management&Monitoring, APM) data, the existing random method will

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] Figure 1a It is a flowchart of a data processing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of performing data processing on the business data in the business table. The method can be executed by the data processing device provided in the embodiment of the present invention. The device can be implemented in the form of software and / or hardware, and can generally be integrated in computer equipment. Such as Figure 1a As shown, the method of this embodiment specifically includes:

[0037] Step 101. Obtain a data writing request, and the data writing request carries a service table.

[0038] Wherein, the data writing request is a request for uniformly storing the business table. Business tables are two-dimensional tables that contain business data. One type of business data corresponds to one business table. Business data is stored column by column in each business column of the business table.

[0039] Figu...

Embodiment 2

[0079] figure 2 It is a flowchart of a data processing method provided by Embodiment 2 of the present invention. This embodiment can be combined with each optional solution in one or more of the above embodiments. In this embodiment, saving the business table to a matching data source according to a preset data distribution strategy may include: according to a preset The data allocation strategy determines the data source that matches the business table in the data write request; creates a data source that matches the business table in the preset relational database, and saves the business table to the matching data source.

[0080] And, according to the data source and the physical table in the preset data storage database, generating data source information and saving it to the data source information list in the preset relational database may include: creating in the preset data storage database The physical table contains a set number of first-type physical storage colum...

Embodiment 3

[0101] image 3 It is a schematic structural diagram of a data processing device provided by Embodiment 3 of the present invention. Such as image 3 As shown, the device can be configured in a computer device, including: a write request acquisition module 301 , a business table storage module 302 , an information generation module 303 , a business column judgment module 304 and a first data writing module 305 .

[0102] Among them, the writing request acquisition module 301 is used to obtain the data writing request, and the data writing request carries the business table; the business table saving module 302 is used to save the business table to the matching data according to the preset data distribution strategy In the source, the data source is located in a preset relational database; the information generating module 303 is configured to generate data source information and save it in the preset relational database according to the data source and the physical table in th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a storage medium. The method comprises the steps of obtaining a data writing request carrying a service table; storing the service table into a matched data source according to a preset data distribution strategy; generating data source information according to the data source and a preset physical table in a data storage database; judging whether a target service column with the same field name as the service column in the service table exists in the service columns in the physical table or not according to a mapping relation list between the service table and the physical table; and if so, generating a mapping relationship between the service table matched with the service table and the physicaltable according to the table column information of the service table, the table column information of the target service column and the table column information of the physical table, and writing theservice data in the service table into the physical table according to the mapping relationship. According to the embodiment of the invention, the compression ratio of the physical storage column canbe improved, the size of the compressed physical storage column is reduced, and the data reading efficiency during data query is improved.

Description

technical field [0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, device, device, and storage medium. Background technique [0002] In most companies, companies have their own independent product lines, and each product will generate many types of two-dimensional table structured data. These structured data are usually stored on a unified storage and provide query functions at the same time. When there are many types of business data and the magnitude is large, distributed columnar storage with compression is often used: use one physical table to store data in multiple business tables, and a column of data in multiple business tables can be stored in the same physical table on a column of . [0003] In the prior art, when multiple business tables are stored in the same table, the physical storage column is basically selected randomly. This method will cause business columns of different d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F16/28
CPCG06F16/2282G06F16/284
Inventor 程捷罗俊刘林
Owner 北京博睿宏远数据科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap