Shuffle data reading method and apparatus, and distributed computing system

By obtaining index files from the Shuffle reader for precise filtering and asynchronous preloading from the remote Shuffle service, the problems of invalid network transmission and synchronous disk I/O in the remote Shuffle service are solved, thereby improving data reading efficiency and computational job performance.

CN122111331APending Publication Date: 2026-05-29BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN Β· China
Patent Type
Applications(China)
Current Assignee / Owner
BEIJING QIYI CENTURY SCI & TECH CO LTD
Filing Date
2026-01-23
Publication Date
2026-05-29

AI Technical Summary

Technical Problem

In the existing remote Shuffle service, there are issues such as invalid network transmission caused by downstream tasks pulling the entire data partition, and high data access latency caused by triggering synchronous disk I/O for each read.

Method used

By obtaining the index file of the data partition on the Shuffle reader, filtering the required data blocks based on preset selection conditions, and sending a preload command to the remote Shuffle service, the system triggers asynchronous loading from disk to memory, thereby achieving precise data filtering and asynchronous preloading.

Benefits of technology

It significantly reduces network data transfer volume, avoids high-latency disk I/O wait, and improves the throughput efficiency of Shuffle data reading and the execution performance of distributed computing jobs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122111331A_ABST
    Figure CN122111331A_ABST
Patent Text Reader

Abstract

The application relates to a Shuffle data reading method and device and a distributed computing system. The method is applied to a Shuffle reader end in a distributed computing system and comprises the following steps: in response to a data reading request for a target data partition, an index file of the target data partition is acquired from a remote Shuffle service, the index file containing position information of each data block in a data file corresponding to the target data partition and associated metadata; the index file is parsed, required data blocks are selected from the index file based on preset selection conditions, and target position information of the required data blocks in the data file is determined; according to the target position information, a preloading instruction is sent to the remote Shuffle service, the remote Shuffle service is instructed to pre-load data corresponding to the target position information in the data file to the memory; and the preloaded data corresponding to the target position information is acquired from the memory of the remote Shuffle service. Therefore, the throughput efficiency of Shuffle data reading and the execution performance of a distributed computing task are greatly improved.
Need to check novelty before this filing date? Find Prior Art