Data de-tilting method and device, electronic equipment and storage medium

A de-slope and data sub-technology, applied in the field of resource optimization, can solve the problems of prolonging the time-consuming of task calculation and slowing down the calculation rate of computing nodes, so as to avoid the single-point calculation problem and reduce the calculation time-consuming effect.

Pending Publication Date: 2022-02-18
BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Computing nodes with a large amount of task data usually take longer to complete the task processing, while other nodes have to wait for the nodes that have not completed the task after pro...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data de-tilting method and device, electronic equipment and storage medium
  • Data de-tilting method and device, electronic equipment and storage medium
  • Data de-tilting method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0078] Aiming at the single-point calculation problem caused by data skew, in the traditional technology, the following two methods are used to solve it:

[0079] 1. Add reduce jvm (Java Virtual Machine, Java virtual machine) memory to the device to optimize data skew (that is, use the increased jvm memory to assist computing nodes with large task processing); however, there are corresponding limitations in this processing method , if there is a memory limit (limited by the availabl...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data de-tilting method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the scattering of a data subset corresponding to a key with a relatively large data size, and scattering the data subset into different tasks, finally, distributing different tasks corresponding to different keys of batch data to a plurality of computing nodes in a relatively balanced manner to be executed so as to realize task configuration (such as configuration as one task or scattering into a plurality of tasks) and computing resource adaptation on the keys based on the data volume of the data corresponding to the keys; and in other aspects, the large-data-volume data corresponding to the keys are scattered, so that large-data-volume tasks do not exist any more, the processing amount of the tasks distributed on each computing node is relatively balanced, data de-inclination is realized, resources of equipment can be fully utilized to perform data computing. The calculation time consumption of the whole task of the batch data is reduced, and the single-point calculation problem caused by data skew is avoided.

Description

technical field [0001] The present application belongs to the technical field of resource optimization, and in particular relates to a data de-skewing method, device, electronic equipment and storage medium. Background technique [0002] Data skew means that in the task allocation and execution of batch data, the dispersion of task data is not enough, and the amount of task data corresponding to different computing nodes is unbalanced. The huge amount of task data of one (some) computing nodes needs to bear huge pressure. And another (some) computing nodes have less task data volume. [0003] Computing nodes with a large amount of task data usually take longer to complete the task processing, while other nodes have to wait for the nodes that have not completed the task after processing the task, which leads to a single-point calculation problem and slows down all tasks. The overall task calculation rate of the calculation node extends the overall calculation time of the tas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
CPCG06F9/5077
Inventor 李虎
Owner BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap