Data skew correction method and device, computer equipment and storage medium

A tilt correction and computer program technology, applied in the computer field, can solve problems such as data tilt and memory overflow that cannot be effectively solved, and achieve the effect of solving data tilt and ensuring uniform distribution

Pending Publication Date: 2021-12-03
紫金诚征信有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, when the amount of data is large, it will still cause memory overflow, which cannot effectively solve the problem of data skew

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data skew correction method and device, computer equipment and storage medium
  • Data skew correction method and device, computer equipment and storage medium
  • Data skew correction method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0040] It should be noted that the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to expressly instead of those steps or elements listed, may include other steps or eleme...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data skew correction method and device, computer equipment and a storage medium. The data skew correction method comprises the following steps: acquiring a data skew correction instruction, and acquiring an original data table from a plurality of servers according to the data skew correction instruction; adding a random sequence in a grouping field of the original data table to obtain intermediate data, and storing the intermediate data in an intermediate table; the column values of the random number columns being non-repeated random numbers for circulation; and determining an association key of the intermediate table, and performing data skew correction according to the association key of the intermediate table. The problem of data skew can be effectively solved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular, to a data tilt correction method, device, computer equipment and storage medium. Background technique [0002] Data skew is a problem often encountered in big data processing. Data skew means that when calculating data, the dispersion of data is not enough, resulting in a large amount of data being concentrated on one or several server nodes for calculation. The calculation of these data The speed is much lower than the average calculation speed, causing the entire calculation process to be too slow. Most big data products solve the problem of data skew by adding physical resources, manually adjusting shuffle parallelism parameters, and partitioning to solve the problem of data skew. However, when the amount of data is large, it will still cause memory overflow, which cannot effectively solve the problem of data skew. Contents of the invention [0003] The main purp...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/21G06F16/22G06F16/27
CPCG06F16/217G06F16/2219G06F16/2282G06F16/27
Inventor 王锦胤路长青
Owner 紫金诚征信有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap