Distributed data processing method and apparatus, electronic device, and storage medium

CN122173566APending Publication Date: 2026-06-09BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
Filing Date
2024-12-06
Publication Date
2026-06-09

AI Technical Summary

Technical Problem

In existing EC technology solutions, offline EC solutions require two storage systems or clusters, resulting in severe amplification of disk and network I/O and poor performance; real-time EC solutions on the SDK side consume memory and CPU resources during EC encoding and decoding, affecting the stability of front-end services and making expansion and maintenance difficult.

Method used

In the target distributed storage system, erasure coding, decoding, and data deletion operations are performed through the target data broker module. A consistent hash ring is used to map the data broker module and the writable storage unit, which avoids resource consumption on the SDK side and reduces IO amplification.

Benefits of technology

It improves the stability of front-end business, avoids amplification of disk and network I/O, reduces the consumption of memory and CPU resources, simplifies expansion and maintenance, and solves the performance bottlenecks and resource consumption problems existing in the technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122173566A_ABST
    Figure CN122173566A_ABST
Patent Text Reader

Abstract

This application provides a distributed data processing method, apparatus, electronic device, and storage medium. The method includes: acquiring a data processing request for a target distributed storage system; identifying a target storage unit in the target distributed storage system that satisfies the data processing request; executing a target operation corresponding to the data processing request on the data service module corresponding to the target storage unit through a target data proxy module, wherein the target operation includes one of the following: erasure coding, erasure coding decoding, and data deletion; and responding to the data processing request based on the result of the target operation. This overcomes the following problems existing in related technologies: offline EC solutions have redundant system architectures, requiring two storage systems or clusters, resulting in severe disk and network I / O amplification; they consume physical disk and network bandwidth, leading to poor performance; and in SDK-side real-time EC solutions, the consumption of memory, CPU, and other resources during EC encoding and decoding can affect the stability of front-end services.
Need to check novelty before this filing date? Find Prior Art