Container-based distributed computing method and device

Abstract

The invention discloses a container-based distributed computing method and device, and the method comprises the following steps in a Shuffle stage: calling a bottom driver to initialize a persistent memory connected to a host machine, determining an equipment application mode for the persistent memory, and creating an area and a namespace on the host machine; creating a data volume with a file system for the persistent memory based on the region and the namespace, and mounting the data volume to a host machine to allow the container to access the file system through a container storage interface; and monitoring the Shuffle management interface, determining Shuffle data according to a dependency relationship between the elastic distributed data sets output by the management interface, and accessing the data volume from the container through the container storage interface so as to overwrite and/or cache the Shuffle data to the file system. According to the method, the processing amountof the Shuffle data can be reduced, the reading and writing cache speed of the Shuffle data in the container can be increased, and then the Shuffle efficiency of distributed computing is improved.

Description

technical field [0001] The present invention relates to the field of distributed computing, and more specifically, to a container-based distributed computing method and device. Background technique [0002] In the Shuffle of the current mainstream distributed memory computing framework, on the one hand, operations such as sorting, joining, and grouping will generate a large amount of Shuffle data when writing code; , SSD, etc.), but the read and write performance of HHD is not good, even the SSD with relatively good read and write performance is far behind the read and write performance of memory, which makes the Shuffle stage consume a lot of time. For Shuffle-intensive computing tasks Very unfriendly. Persistent memory has read and write performance close to that of DRAM and has a capacity that cannot be compared with ordinary DRAM. However, there are certain technical barriers in the interaction between the container and the host's persistent memory device, and the conta...

AI Technical Summary

Problems solved by technology

[0002] In the Shuffle of the current mainstream distributed memory computing framework, on the one hand, operations such as sorting, joining, and grouping will generate a large amount of Shuffle data when writing code; , SSD, etc.), but the read and write performance of HHD is not good, even the SSD with relatively good read and write performance is far behind the read and write performance of memory, which makes the Shuffle stage consume a lot of time. For Shuffle-intensive computing tasks very unfriendly

Persistent memory has read and write performance close to that of DRAM and has a capacity that cannot be compared with ordinary DRAM. However, there are certain technical barriers in the interaction between the container and the host's persistent memory device, and the container cannot recognize the persistent storage on the host. equipment

[0003] There is currently no effective solution to the problems of excessive Shuffle data volume and slow reading and writing cache speed in the container in the existing technology

Images