Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device, computer equipment and storage medium

A technology for data processing and computer programs, applied in the field of non-transitory computer-readable storage media and computer program products, can solve problems such as uneven data distribution, data skew, and many read/write requests, achieve uniform data distribution, eliminate The effect of skewing data, ensuring accuracy and reliability

Pending Publication Date: 2021-06-04
北京中经惠众科技有限公司
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In a big data computing cluster, due to the uneven distribution of key values, the characteristics of the business data itself, or poor consideration when creating tables, etc., data skew may occur, that is, the data distribution of each node in the computing cluster is uneven.
This will lead to excessive read / write requests, heavy load, and long computing time for some nodes, which will affect the overall computing speed of the big data cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, computer equipment and storage medium
  • Data processing method and device, computer equipment and storage medium
  • Data processing method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] In the present disclosure, unless otherwise stated, using the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship or importance relationship of these elements, and such terms are only used for Distinguishes one element from another. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on contextual description.

[0021] The terminology used in describing the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, there may be one or more elements. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". In ad...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium. The method comprises the following steps: respectively splitting a first data set to be connected and a second data set to be connected into a plurality of first partitions and a plurality of second partitions; determining the data volume of each of the plurality of first partitions and the plurality of second partitions; according to the connection type of the first data set and the second data set and the determined data volume of each partition, selectively re-splitting the plurality of first partitions and the plurality of second partitions to obtain a plurality of first data set partitions and a plurality of second data set partitions; and allocating the plurality of first data set partitions and the plurality of second data set partitions to respective computing nodes for connection of the first data set and the second data set.

Description

technical field [0001] The present disclosure relates to the technical field of big data and data processing, and in particular, to a data processing method, device, computer equipment, non-transitory computer-readable storage medium, and computer program product. Background technique [0002] Big data refers to a collection of data that is so large that it exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It has massive data scale, fast data flow, diverse data types, and low value density. Four characteristics. Faced with such large-scale data, it is necessary to process, analyze and aggregate it through big data computing clusters to find useful information from these data, so as to provide services for upper-level applications and provide decision-making for users. [0003] In a big data computing cluster, due to the uneven distribution of key values, the characteristics of the business data itself...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/22G06F16/27
CPCG06F16/2282G06F16/278
Inventor 向鹏杨令卿黄江
Owner 北京中经惠众科技有限公司