Key value data processing method and device, electronic equipment and storage medium

A processing method and technology of electronic equipment, applied in the computer field, can solve problems such as the inability to meet the key-value data sorting requirements of MapReduce jobs, and achieve the effects of easy deployment and maintenance, reduced modification, and high versatility

Pending Publication Date: 2022-03-25
BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, most of the current stream shuffling mechanisms within enterprises only support the Spark engine.
When there are both Spark and MapReduce jobs in the enterprise, the hash-based shuffling method of the Spark engine cannot meet the key-value data sorting requirements of the MapReduce job. Therefore, a situation that can work on multiple computing engines at the same time is urgently needed Next, the way to flexibly sort the key-value data of various computing engines

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Key value data processing method and device, electronic equipment and storage medium
  • Key value data processing method and device, electronic equipment and storage medium
  • Key value data processing method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0078] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0079] It should be noted that the terms "first" and "second" in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent w...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a key value data processing method and device, electronic equipment and a storage medium. The method comprises the steps that a to-be-processed task and meta information corresponding to the to-be-processed task are acquired; the to-be-processed task is processed through the mapping task in the calculation engine to obtain key value data, the key value data are transmitted to shuffling processing nodes, and the shuffling processing nodes are nodes independently packaged outside the calculation engine; and running sorting logic corresponding to the meta-information through the shuffling processing node so as to sort the key value data. According to the scheme provided by the invention, the key value data generated by various calculation engines are processed by adopting a general sorting mechanism, so that the sorting requirements of the key value data of the various calculation engines can be met at the same time.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic equipment, computer-readable storage medium, and computer program product for processing key-value data. Background technique [0002] The current computing engine Mapreduce (a map-reduce model), Spark (a computing engine), etc. have a shuffle (shuffling) mechanism. The shuffling mechanism is responsible for the data transmission between MapTask (mapping task) and ReduceTask (reduction task) in the task. [0003] Currently, the shuffling mechanism is mainly divided into sort-based shuffle (sort-based shuffling) and hash-based shuffle (hash-based shuffling). Mapreduce mainly adopts a sorting-based shuffling method, which sorts key-value (key-value) data on the mapping task side according to partition (partition) and key (key), and outputs them to the local disk to form an orderly shuffling Wash files. In this way, the shuffled files o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F16/2458
CPCG06F9/5038G06F16/2474
Inventor 李超
Owner BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap