Data processing method and device
A data processing and data table technology, applied in the field of data processing, can solve problems such as low data processing efficiency, reduced performance of distributed storage systems, data copy data operations, etc.
Pending Publication Date: 2022-05-20
HUAWEI TECH CO LTD
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In this process, when the key included in the query statement is the same as the pre-specified distribution key, the data copy to be operated can be located directly through the distribution key, and then the data included in the data copy can be selected according to the query conditions in the query statement In the sharding, determine the data shards that need to perform data operations to perform data operations on the data shards; when the key included in the query statement is different from the pre-specified distribution key, it cannot be pre-created directly based on the distribution key pair However, some additional operations are required. For example, it is necessary to scan the entire table, broadcast the table, or redistribute data to determine the data that needs to be operated. In this way, the data processing efficiency is low and the distributed storage system performance
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0052] In order to make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0053] The distributed storage system adopts an expandable system structure, uses multiple storage servers to share the storage load, and uses location servers to locate and store information. It not only improves the reliability, availability and access efficiency of the system, but is also easy to expand.
[0054] A distributed database is deployed in the distributed storage system, that is to say, the data in the distributed database is distributed and stored in the distributed storage system, and the distributed storage system can be understood as a distributed cluster. There are multiple servers deployed in the cluster, and the data in the data table can be distributed and stored across multiple servers. There are ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a data processing method and device which are used for improving data processing efficiency. The method comprises the steps that a data operation request is obtained, the data operation request comprises a target key, and the data operation request is used for requesting data operation on a target data table; selecting a target data copy from at least two data copies corresponding to the target data table according to the target key; and sending a data operation instruction for performing data operation on the target data copy. According to the method, the number of the data copies is increased, so that the probability of directly selecting the target data copies according to the target keys is naturally increased, data operation can be quickly performed on the target data copies, the data operation efficiency for the target data table is improved, and the data processing efficiency is further improved.
Description
technical field [0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device. Background technique [0002] In a distributed storage system (such as greenplum database), a distribution key is generally specified when creating a data table, and then the data in the data table can be fragmented according to the specified distribution key to obtain a data copy, and then the data copy includes Each data fragment is stored in different data nodes in the distributed storage cluster to realize distributed storage of data. [0003] In the actual data business, the data operation on the data table is generally performed through the query statement. The query statement generally includes the key in the data table (that is, the column name of the data table). The system judges the key in the query statement and the specified Whether the distribution key matches determines whether data operations can be perfo...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/22G06F16/23G06F16/27G06F11/14
CPCG06F16/2228G06F16/23G06F16/27G06F11/1448G06F11/1464
Inventor 刘伟胡翔宇崔岩
Owner HUAWEI TECH CO LTD