Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device of distributed assembly line and storage medium

A data processing and pipeline technology, applied in the database field, can solve problems such as waste, and achieve the effect of increasing the amount of data sent, reducing the amount of data storage, and reducing the time-consuming of query execution.

Pending Publication Date: 2022-05-03
PINGCAP XINGCHEN (BEIJING) TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

While upstream and downstream subtasks are waiting for network transmission, CPU resources can only be idle, causing waste

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device of distributed assembly line and storage medium
  • Data processing method and device of distributed assembly line and storage medium
  • Data processing method and device of distributed assembly line and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, but not to be construed as a limitation on the present application.

[0029] It will be understood by those skilled in the art that the singular forms "a," "an," and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operatio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data processing method and device of a distributed assembly line, electronic equipment and a storage medium, relates to the technical field of databases, and is suitable for a cross-node multi-task data exchange scene in an MPP database. The method comprises the following steps: creating a data processing thread and a data transmission thread which are asynchronously executed; obtaining compressed data generated by the upstream task through a data transmission thread; decompressing the compressed data of the upstream task through a data processing thread, executing the current task, determining and compressing the data generated by the current task according to a current target compression algorithm, and storing the compressed data generated by the current task into a sending buffer area; and sending the data in the sending buffer area to a corresponding downstream task through a data transmission thread. According to the embodiment of the invention, the computing power of the nodes can be fully utilized, dynamic balance of production data and sending data is realized, the throughput and resource utilization rate of the whole assembly line are improved, and query execution time consumption is reduced.

Description

technical field [0001] The present application relates to the technical field of databases, and in particular, the present application relates to a data processing method, device and storage medium of a distributed pipeline. Background technique [0002] In order to analyze big data in real time and extract the value of big data, Massive Parallel Processing (MPP) database systems are widely used, such as SparkSQL, Impala, Greenplum and so on. The MPP database system runs in a cluster, and a cluster includes multiple physical machines and is connected through a network. In order to make full use of CPU and network resources to improve performance, many MPP databases use distributed pipeline technology to process tasks. These tasks involve the cooperation of multiple machines across the network. Every time an upstream subtask generates a small piece of data, it is sent to the downstream subtask for processing through the network. Distributed pipeline technology utilizes both...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/24G06N3/02H03M7/30
CPCG06F16/24G06N3/02H03M7/30
Inventor 方祝和刘奇黄东旭崔秋
Owner PINGCAP XINGCHEN (BEIJING) TECH CO LTD