Data processing method and device

By adjusting the data granularity in a high-performance computing system to adapt to the storage cluster attributes, the problem of data read and write performance bottlenecks is solved, and automatic optimization of application IO performance and efficient utilization of storage clusters are realized.

CN119739472APending Publication Date: 2025-04-01HUAWEI TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202311291628.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-09-29
Publication Date
2025-04-01

AI Technical Summary

Technical Problem

In high-performance computing systems, data read and write performance often becomes a performance bottleneck, and it is difficult for the prior art to effectively optimize the IO performance of applications to adapt to the hardware architecture evolution of storage clusters.

Method used

Provide a data processing method, by obtaining the data to be written by the application process, determining the granularity of the data according to the attributes of the storage cluster, and adjusting the data to be written to a suitable size, so as to optimize the IO operation and realize IO optimization that automatically adapts to the storage cluster.

Benefits of technology

Without modifying application code, significantly improve the IO writing performance of the application, ensure that IO operations run within the performance range of the back-end storage cluster, and improve the operation performance of large-scale parallel studies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119739472A_ABST
    Figure CN119739472A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and device, the method is applied to a computing cluster, the computing cluster comprises a plurality of computing nodes, the computing cluster is connected to a storage cluster, the storage cluster is used for storing data generated by the computing cluster, and a plurality of processes of an application run in the computing nodes. Each process executes a part of data of the application, and the method comprises the following steps: obtaining N pieces of first to-be-written data of the plurality of processes; adjusting the N pieces of first data to be written into M pieces of second data to be written according to a first granularity of data accessing the storage cluster, the size of each piece of second data to be written being the first granularity, and the first granularity being determined according to the attribute of the storage cluster; and writing the M second data to be written into the storage cluster.
Need to check novelty before this filing date? Find Prior Art

Citation Information

Cited By

  • Data processing method and device

    EP4776129A1