Data processing method and device

Abstract

A data processing method and device. The method is applied to a computing cluster, wherein the computing cluster comprises a plurality of computing nodes and is connected to a storage cluster, the storage cluster is used for storing data generated by the computing cluster, a plurality of processes of an application run in the plurality of computing nodes, and each process is used for executing part of the data of the application. The method comprises: acquiring N pieces of first data to be written of a plurality of processes; on the basis of a first granularity for access of data in a storage cluster, adjusting said N pieces of first data to be M pieces of second data to be written, the size of each of said pieces of second data being the first granularity, and the first granularity being determined on the basis of attributes of the storage cluster; and writing said M pieces of second data to the storage cluster.

Description

Data processing method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on September 29, 2023, with application number 202311291628.7 and application name "A Data Processing Method and Device", the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of high performance computing technology, and in particular to a data processing method and device. Background Art

[0004] In large-scale computing systems such as high-performance computing (HPC) and supercomputing (SC), data read and write performance has a crucial impact on the performance of applications running on large-scale supercomputer clusters. Data read and write often becomes a performance bottleneck for large-scale workloads. Currently, there is an urgent need for a solution to optimize the data read and write performance of applications.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a data processing method and apparatus for providing an IO optimization method that automatically adapts to a storage cluster and improves the IO performance of an application.

[0007] In a first aspect, the present application provides a data processing method, which can be applied to a computing cluster, wherein the computing cluster includes multiple computing nodes, the computing cluster is connected to a storage cluster, the storage cluster is used to store data generated by the computing cluster, and multiple processes of an application run in the multiple computing nodes, and each process executes a portion of the data of the application. The method includes: obtaining N first data to be written of the multiple processes of the application; determining a first granularity for accessing the data of the storage cluster according to the attributes of the storage cluster, and adjusting the N first data to be written into M second data to be written according to the first granularity, and the size of each second data to be written is the first granularity; and writing the M second data to be written into the storage cluster.

[0008] Through the above method, N first data to be written generated by multiple application processes are obtained. A first granularity for accessing the data in the storage cluster is determined based on the properties of the storage cluster. The N first data to be written are reorganized according to the first granularity, and the reorganized M second data to be written are written to the storage cluster. This approach automatically optimizes the application's I / O operations to adapt to the storage cluster by combining the properties of the storage cluster without modifying the application's program code. This completely decouples the application logic from the underlying I / O operations. This allows the adjusted I / O operations to be controlled within a performance range that is optimal for the backend storage cluster, significantly improving the application's actual write I / O performance.

[0009] In one possible design, the N first data to be written are adjusted to M second data to be written according to the first granularity of the data accessed in the storage cluster, including: splitting the first data to be written that is larger than the first granularity among the N first data to be written according to the first granularity; and / or splicing the first data to be written that is smaller than the first granularity among the N first data to be written into the first granularity.

[0010] Through the above method, multiple IO operation optimization methods and flexibility are provided.

[0011] In one possible design, the method further includes: when M is greater than a preset value, adjusting the size of the first granularity until M is less than or equal to the preset value.

[0012] The above method provides flexibility in IO optimization.

[0013] In one possible design, the method further includes: determining multiple first data to be read of the multiple processes; obtaining at least one second data to be read from the storage cluster according to a second granularity of accessing data of the storage cluster, the at least one second data to be read including the multiple first data to be read, the size of each second data to be read being the second granularity, and the second granularity being determined according to the attributes of the storage cluster.

[0014] In one possible design, the second granularity for accessing the storage cluster is determined based on the properties of the storage cluster, and data is read from the storage cluster based on the second granularity. This approach can automatically optimize the application's IO operations to adapt to the storage cluster based on the properties of the storage cluster without changing the application's program code, completely decoupling the application logic from the underlying IO operations. In this way, the adjusted IO operations can be controlled within a performance range that is optimal for the back-end storage cluster, significantly improving the application's actual read IO performance.

[0015] In the second aspect, the present application also provides a data processing device, which has the function of implementing the behavior in the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the device includes an acquisition module, a processing module, an execution module, and optionally, an adjustment module and a determination module; these modules can implement the functions of the behavior in the method of the first aspect above. For details, please refer to the detailed description in the method example and will not be repeated here.

[0016] In a third aspect, the present application further provides a computing device. The computing device comprises a processor and a memory, wherein the memory is used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes program instructions stored in the memory, the method described in the first aspect can be implemented. The computing device also includes a communication interface for communicating with other devices, such as obtaining data corresponding to a task from first-tier storage resources.

[0017] In a fourth aspect, the present application further provides a data processing system, which includes at least one computing device, and the at least one computing device has the function of implementing the behavior in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The structure of each computing device includes a processor and a memory, and the processor is configured to support the computing device to perform part or all of the functions of the behavior in the first aspect and any possible implementation of the first aspect. The memory is coupled to the processor, which stores the necessary program instructions and data for the computing device. The structure of the computing device also includes a communication interface for communicating with other devices.

[0018] In a fifth aspect, the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0019] In a sixth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0020] In the seventh aspect, the present application also provides a computing chip, which is connected to a memory and is used to read and execute software programs stored in the memory, and to execute the methods in the above-mentioned first aspect and various possible implementations of the first aspect.

[0021] For the description of the beneficial effects of any of the second to seventh aspects mentioned above, please refer to the description of the technical effects in any possible implementation method of the first aspect mentioned above, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of a system architecture that may be applicable according to an embodiment of the present application;

[0023] FIG2 is a schematic diagram of multiple tasks of an operation provided by an embodiment of the present application;

[0024] FIG3 is a schematic diagram of the first IO mode;

[0025] FIG4 is a schematic diagram of the second IO mode;

[0026] FIG5 is a schematic diagram of the third IO mode;

[0027] FIG6 is a schematic diagram of the fourth IO mode;

[0028] FIG7 is a flow chart of a data processing method according to an embodiment of the present application;

[0029] FIG8 is a process diagram of a data processing method provided in an embodiment of the present application;

[0030] FIG9 is a schematic diagram of the structure of an IO module provided in an embodiment of the present application;

[0031] FIG10 is a schematic diagram showing a comparison of data before and after reorganization according to an embodiment of the present application;

[0032] FIG11 is a schematic diagram of IO operation modes before and after IO optimization provided in an embodiment of the present application;

[0033] FIG12 is a schematic diagram of the structure of a storage and processing device provided in an embodiment of the present application;

[0034] FIG13 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] High-performance computing (HPC) is a cross-industry and cross-application computing discipline that typically applies cutting-edge computer technologies to the most complex and cutting-edge scientific calculations and solutions. It is widely used in computing large-scale scientific problems and processing massive amounts of data, such as weather forecasting, earthquake prediction, automotive simulation, biopharmaceuticals, gene sequencing, nuclear explosion simulation, rocket launch simulation, and chip design and manufacturing. A computer cluster capable of providing HPC services is called an "HPC cluster."

[0036] A computer cluster is a group of loosely or tightly connected computing nodes working together, typically to execute large jobs. Cluster deployments typically improve overall performance through concurrency and are more cost-effective than single computing nodes of comparable speed or availability. Each computing node is connected to the other via a network and runs its own operating system instance. In most cases, each computing node uses the same hardware and the same operating system; in some cases, different operating systems can be used on different hardware.

[0037] Figure 1 is a schematic diagram of a system architecture that may be applicable to an embodiment of the present application. It should be understood that the system architecture is merely an exemplary description, which only provides components related to the embodiment of the present invention, and the number of nodes shown in each component is also merely an example. However, Figure 1 does not specifically limit the application of the present invention.

[0038] As shown in Figure 1, computing cluster 10 includes multiple computing nodes, such as 100A, 100B, 100C, 100D, and 100E. These computing nodes are used to provide computing resources. At the software level, applications can be installed and run on top of the operating system for each computing node. Applications are a shorthand term for a series of application programs, including those used to execute jobs, such as HPC applications, artificial intelligence (AI) applications, and big data processing applications. HPC applications specifically include WRF (used for weather forecasting) and OpenFOAM (used in aerodynamics), and different applications can be used to process different types of jobs. At the hardware level, it can include multiple processors or processor cores. Each processor may include one or more processor cores, and each processor or processor core may be a computing resource. Therefore, a single physical computing node can provide multiple computing resources. Computing nodes 100A, 100B, 100C, 100D, and 100E are interconnected via network 112. Furthermore, computing node 160, acting as a scheduler, is also connected to network 112. In operation, scheduler 160 may control the execution of jobs submitted to compute cluster 10 .

[0039] Jobs can be submitted to the computing cluster 10 from any suitable source. The embodiments of the present application do not limit the location of submitting jobs, nor the specific mechanism by which users submit jobs. For example, in Figure 1, user 132 can submit job 136 to the computing cluster 10 from enterprise 130. Specifically, in this example, user 132 operates client computer 134 to submit job 136 to the computing cluster 10. In this instance, enterprise 130 is connected to computing cluster 10 via network 120, which generally represents any telecommunications or computer network, including, for example, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), or the Internet. Therefore, user 132 can submit job 136 to the computing cluster 10 from a remote location.

[0040] In the present application, when submitting a job 136, the user 132 may also operate the client computer 134 to submit additional information of the job 136, such as the data and configuration information of the job 136. The configuration information includes, but is not limited to, the number of computing nodes, the number of processor cores, the number of processes, etc. used to execute the job 136. Accordingly, the scheduler 160 may be used to provide a user interface for submitting jobs, manage computing resources (such as monitoring the busy or idle status of computing resources), control resource allocation and lifecycle management of application execution, such as scheduling the job 136 to multiple idle computing resources, controlling computing resources to start applications to run the job 136, controlling computing resources to stop applications to terminate the job 136, and so on.

[0041] Job 136 is typically a large job that requires multiple computing resources to be processed in parallel. This embodiment does not limit the nature and number of jobs. A job may include multiple computing tasks (referred to as tasks), which can be assigned to multiple computing resources for execution. Most tasks are executed concurrently or in parallel, while some tasks require data generated by other tasks.

[0042] For example, in conjunction with Figure 2, assume that job 136 is to predict the weather conditions of city A in the next 24 hours. City A includes multiple areas, which are respectively recorded as area 1, area 2,..., area n (n is a positive integer). Exemplarily, job 136 is divided into multiple tasks based on time and area granularity. As shown in Figure 2, the prediction of the weather conditions of city A in 24 hours is divided into 24 rounds of iterations (steps) with 1 hour as the unit. Each round of iteration is used to predict the weather conditions of different areas of city A within the corresponding 1 hour, thereby splitting the job 136 into parallel tasks executed in multiple rounds of iterations. For example, the first round of iteration includes tasks 1 to n executed in parallel, where task 1 is used to predict the weather conditions of area 1 from 0:00 to 1:00 in the future, task 2 is used to predict the weather conditions of area 2 from 0:00 to 1:00 in the future, task 3 is used to predict the weather conditions of area 3 from 0:00 to 1:00 in the future, and so on. Similarly, the second round of iteration includes tasks 1' to n' executed in parallel, where task 1' is used to predict the weather conditions of area 1 from 1:00 to 2:00 in the future, task 2' is used to predict the weather conditions of area 2 from 1:00 to 2:00 in the future, task 3' is used to predict the weather conditions of area 3 from 1:00 to 2:00 in the future, and so on.

[0043] Taking a round of iteration as an example, multiple tasks of a round of iteration can be scheduled to idle computing resources in the computing cluster. Each computing resource can run one or more processes of the application. In Figure 2, the multiple tasks included in a round of iteration in job 136 can be assigned to be executed by multiple parallel processes, such as process 1 to process n are used to execute tasks 1 to task n respectively, and each process is used to execute one task.

[0044] Among them, the process of executing a task for each process may be the same. Simply put, the process includes performing calculations based on the input data of a task to obtain the output result of the task. Taking Task 1 as an example, the execution process of Task 1 specifically includes: Process 1 obtains the input data of Task 1 from the storage cluster 20 through one or more read IOs. The input data of Task 1 may include the meteorological data of Region 1 within a time period, such as temperature, wind speed, wind direction, air humidity, rainfall, etc. Based on the input data, calculations are performed (for weather forecast tasks, equations can be constructed through mathematical methods, and the meteorological data of the region within a period of time can be imported into the same equation for calculation and solution, thereby predicting the atmospheric changes and status of the region within a period of time in the future) to obtain the output result of Task 1, such as the output result including the meteorological data of Region 1 in the future 0:00-1:00. Afterwards, Process 1 can write the output result of Task 1 to the storage cluster 20 through one or more write IOs.

[0045] The storage cluster 20 is communicatively connected to the computing cluster 10. The storage cluster 20 may include multiple storage nodes. The storage devices may be storage servers, storage arrays, etc., and are typically used to persistently store data. Each storage node may include one or more hard disks. The hard disks may be magnetic disks or other types of storage media, such as solid-state drives (SSDs), hard disk drives (HDDs), storage class memory (SCMs), persistent memory (PMs), shingled magnetic recording hard disks, or volatile memory. The specific types are not limited thereto, and any memory capable of storing data is applicable to this embodiment.

[0046] In this embodiment, the storage cluster 20 can be used to store the data of job 136. The data of job 136 includes the initial data of job 136 (such as the input data of multiple tasks of job 136, configuration files, etc.). The initial data of job 136 can be submitted by the user or generated by the application depending on the previous job. The data of job 136 also includes data such as intermediate processing results (such as the output results of the task) generated by the process in the process of executing job 136. In one embodiment, a file system is established in the storage cluster 20. In this case, the data of job 136 is stored in the storage cluster 20 in the form of files. Accordingly, when running the tasks of job 136, the multiple processes of the application read the data of the files in the storage cluster 20 (such as the input data of the task) through one or more read IOs, and write the data to be written (such as the output results of the task) to the corresponding files in the storage cluster 20 through one or more write IOs.

[0047] The way an application controls multiple processes to read and write files is called an IO mode. IO modes are distinguished by the number of processes and files performing IO operations. The following are some examples of IO modes:

[0048] (1) rank 0 mode (N:1)

[0049] As shown in FIG3 , the application includes N processes, which concurrently read and write a single file in the storage cluster 20. Specifically, for this mode, the application can perform IO operations (including data read and write operations) through IO interfaces such as HDF5, PNetCDF, and MultiPoint Interface (MPI)-IO. Such IO interfaces typically create a single file for a large number of application processes on multiple computing nodes to read and write data in parallel. For example, the N processes in FIG3 can be multiple processes used to execute tasks 1 to n in FIG2 . The single file can be used to store data such as the input data and output results of tasks 1 to n. The N processes all read and write data to the single file.

[0050] For the mainstream parallel file systems in existing businesses (such as Lustre, GPFS, etc.), this method cannot fully utilize the total aggregate bandwidth of the parallel file system when handling massive processes concurrently reading and writing the same single large file, resulting in a bottleneck problem.

[0051] (2) Independent IO mode (1:1)

[0052] As shown in FIG4 , each application process independently interacts with the storage cluster 20. Generally, each process accesses its own unique file, and there is no data exchange between processes. Specifically, in this mode, applications can perform IO operations through a portable operating system interface (POSIX) or an underlying system interface. Such an IO interface typically creates one or more files for each process. For example, the N processes in FIG4 are multiple processes used to execute tasks 1 to n in job 136. Each process corresponds to a file, and each file is used to store data such as the input data and output results of the task executed by the corresponding process. The N processes read and write data to their corresponding files.

[0053] The total number of files generated by this approach increases with multiple dimensions, including the number of processes, the number of application iterations, and the number of data sets read and written by the application. In a production environment where large-scale concurrent instances are running, the excessive number of files often leads to parallel file system failures and a sharp drop in actual I / O bandwidth.

[0054] (3) Aggregate IO mode (1:N)

[0055] As shown in Figure 5, multiple processes of the application send requests indicating data to be read or written to one of the processes (denoted as the aggregation process), and the aggregation process performs data aggregation, such as aggregating multiple small blocks of data to be read into a large block of data to be read, and aggregating multiple small blocks of data to be written into a large block of data to be written. Specifically, for the mode, the application can first aggregate data through the MPI communication interface, and then call POSIX, MPI-IO, HDF5, PNetCDF, NetCDF and other IO interfaces to perform IO operations. Exemplarily, the IO operation may include generating a read IO request or a write IO request. The read IO request is used to request to obtain the large block of data to be read from the storage cluster 20, and the write IO request is used to request to write the large block of data to be written to the storage cluster 20.

[0056] This approach requires additional CPU and memory overhead because applications need to perform their own data aggregation. Furthermore, the data aggregation logic is embedded in the application's logic code and cannot evolve with the storage cluster's hardware architecture. As a result, many applications may achieve relatively good performance on the storage cluster used in the early stages of development, but as the storage cluster's hardware architecture evolves, the application's read and write performance gradually deteriorates, eventually forming a bottleneck.

[0057] (4) Aggregate IO mode (M:N)

[0058] As shown in Figure 6, this mode divides the N processes included in an application into m aggregation groups. Each aggregation group includes one aggregation process. This aggregation process has the same functionality as the aggregation process in the Aggregate IO Mode (1:N). The difference is that the Aggregate IO Mode (M:N) has more aggregation processes, and each aggregation process is responsible only for aggregating data within the aggregation group. Specifically, in this mode, the application calls interfaces such as ADIOS to perform IO operations.

[0059] This approach requires applications to proactively adapt to the storage system's hardware architecture by modifying program code. This leads to high development costs for applications with complex I / O logic, preventing most applications from taking advantage of optimization capabilities. Furthermore, during runtime, applications must manually configure parameters such as I / O modes and aggregation logic through configuration files, preventing automatic adaptation to the backend storage cluster. This makes it difficult to achieve optimal performance in complex production environments.

[0060] Currently, applications typically use a fixed I / O model to read and write data. Data read and write performance has a crucial impact on the performance of applications running on large-scale supercomputer clusters. When the I / O model used by an application inherently has poor read and write performance, or when the application's I / O model becomes incompatible with the storage cluster due to storage cluster upgrades, coupled with the rapid evolution of storage cluster hardware architectures and the diverse architecture types, the coding and development workload for large-scale parallel applications is inherently large. In practice, modifying the application's I / O operation logic to adapt to evolving storage cluster hardware architectures is extremely difficult. This makes it difficult for applications to achieve optimal performance in complex production environments, and further, data read and write often becomes a performance bottleneck for large-scale parallel computing instances.

[0061] In view of this, an embodiment of the present application provides a data processing method. This method optimizes the IO operations of the application program by combining the properties of the storage cluster without changing the program code of the application, automatically adapts to the evolution of the storage cluster hardware architecture, improves the IO operation performance of the application, enables the application to automatically obtain higher performance in a complex production environment, and further improves the running performance of large-scale parallel examples.

[0062] The data processing method provided in the embodiment of the present application is described in detail below with reference to FIG1 and FIG2 as examples.

[0063] FIG7 is a flow chart illustrating a data processing method provided in an embodiment of the present application. This method can be applied to the computing cluster 10 shown in FIG1 . The computing cluster 10 includes multiple computing nodes, each of which includes one or more computing resources. Multiple processes included in an application for executing a job run on the multiple computing resources. The computing cluster 10 is connected to a storage cluster 20, which is used to store job data. The method can be executed by multiple computing nodes in the computing cluster 10 for executing jobs, or by components of the multiple computing nodes (e.g., computing resources (e.g., processors or processor cores) for executing tasks). For ease of description, the following description uses job 136 as an example. The method may include:

[0064] Step 701: Obtain N data to be written (recorded as the first data to be written) of multiple processes.

[0065] Taking job 136 as an example, and referring to FIG8 , assume that the application used to process job 136 is running on the multiple computing nodes shown in FIG8 . The application includes multiple processes, each of which is used to execute one or more tasks of job 136. When executing tasks, the multiple processes may generate first data to be written, such as task output results. In this example, step 701 may include obtaining N first data to be written from the multiple processes included in the application used to execute job 136.

[0066] An embodiment of the present application provides an IO module suitable for large-scale parallel applications. The functions of the IO module include obtaining data to be read or written by an application process and writing the data to be written into a storage cluster 20 .

[0067] Next, we will introduce how to obtain the first data to be written in a process in conjunction with the IO module provided in the embodiment of this application:

[0068] FIG9 is a schematic diagram of the structure of an IO module provided in an embodiment of the present application. The IO module may be a software module running in a user process of a computing node. The user process may be a process for executing tasks.

[0069] As shown in Figure 9, the IO module may include but is not limited to some or all functional modules in the interface layer, control layer, orchestration layer, communication layer, transport layer and offline layer. Among them, the interface layer includes bridges (or interceptors) or plug-ins for various IO interfaces, for example, PNetCDF bridge, NetCDF bridge, HDF5 bridge, ADIOS bridge, MPI-IO interceptor, POSIX interceptor. Among them, each IO interface bridge / interceptor corresponds to the original IO interface, for example, the PNetCDF bridge corresponds to the PNetCDF interface, the NetCDF bridge corresponds to the NetCDF interface, and the HDF5 bridge corresponds to the HDF5 interface. Exemplarily, the IO library corresponding to the IO interface bridge / interceptor and the IO library corresponding to the original IO interface may have the same function name and function list, but different function functions. Each IO interface bridge can be used to intercept the process's call to the original IO interface to obtain the data to be written generated by the process or to determine the data to be read by the process.

[0070] For example, taking an application writing data in rank 0 mode as an example, in one implementation, processes 1 through n of the application can each trigger a call to the HDF5 interface. Each process's call includes the corresponding process's first data to be written, which can be used to request that each first data to be written be written to a single file in storage cluster 20, such as file 1. At this point, the IO module within each process intercepts the process's call to the HDF5 interface via the HDF5 bridge to obtain the process's first data to be written. In another implementation, the application process may also trigger a call to the NetCDF interface to write the process's first data to be written to file 1. The IO module within the process can intercept the process's call to the NetCDF interface via the NetCDF bridge to obtain the process's first data to be written.

[0071] Among them, the IO modules in each of the multiple processes of the application can constitute a data IO processing system, wherein each IO module respectively performs the above-mentioned interception operation to obtain N first data to be written of the multiple processes, and the N first data to be written can be generated by N parallel processes respectively, and each process generates one first data to be written. Alternatively, the N first data to be written may also be generated by (NL) processes within a period of time, and each process generates one or more first data to be written. The N first data to be written may be generated in the same round of iteration or in different rounds of iteration, and there is no specific limitation. For the sake of convenience, the following is an example of the N first data to be written being generated by N processes in a round of iteration of executing job 136.

[0072] Step 702 : Determine the granularity of accessing data in the storage cluster 20 (referred to as the first granularity).

[0073] Taking write operations as an example, the first granularity can be the write granularity, which refers to the size of the data block to be written in a write operation. For example, a write granularity of 4KB means that a 4KB data block is written each time, and a write granularity of 8KB means that an 8KB data block is written each time. Similarly, in read operations, the first granularity can be the read granularity, which refers to the size of the data block to be read in a read operation. For example, a read granularity of 4KB means that a 4KB data block is read each time, and a read granularity of 8KB means that an 8KB data block is read each time. This first granularity corresponds to the target performance range of storage cluster 20. When writing data to be written to storage cluster 20 at this first granularity, the application can achieve higher read and write performance.

[0074] How to determine the first granularity is described below.

[0075] In one embodiment, the first granularity is determined based on attribute information of the storage cluster 20, where the attribute information is used to indicate attributes of the storage cluster 20. Exemplarily, the attribute information of the storage cluster 20 may include, but is not limited to, one or more of the following: network topology information of the storage cluster 20 and performance information of the storage cluster.

[0076] The network topology information of the storage cluster 20 is used to indicate the number of storage nodes included in the storage cluster 20 and / or the connection relationships between storage nodes and between storage nodes and network devices such as switches.

[0077] The performance information of the storage cluster 20 includes a correspondence between access parameters of the storage cluster 20 and storage performance. The access parameters may include, but are not limited to, write granularity, read granularity, number of write files, number of read files, etc. In one example, the correspondence includes a correspondence between different write granularities and the storage performance of the storage cluster 20. For example, the correspondence includes the storage performance corresponding to a write granularity of 4KB, the storage performance corresponding to a write granularity of 8KB, and the storage performance corresponding to a write granularity of 16KB. The first granularity may be a value within the write granularity range corresponding to the target performance interval of the correspondence. It may be a randomly selected value, a preset value, or a value determined by other methods. In another example, there is also a correspondence between the number of write files and storage performance. The number of write files may refer to the number of files involved in a parallel write operation by multiple parallel processes. For example, the correspondence includes a correspondence between different numbers of write files and the storage performance of the storage cluster 20. The storage performance of the storage cluster 20 may be determined by one or more performance indicators such as the effective bandwidth, throughput, and input / output per second (IOPS) of the storage cluster 20. The above correspondence relationship may be determined based on an algorithm or by performing multiple tests on the storage cluster 20 , which is not limited in this embodiment.

[0078] The specific implementation method for determining the first granularity is introduced below in conjunction with the IO module.

[0079] Continuing with Figure 9 , the functional modules of the IO module also include a policy generator, a storage hierarchy orchestrator, and a policy generator. The storage hierarchy orchestrator can be used to perceive attribute information of the storage cluster 20, the policy generator can be used to generate IO optimization policies, and the policy manager can include multiple IO optimization methods. The policy manager can be used to select one or more IO optimization methods from the multiple IO optimization methods to generate one or more IO optimization policies applicable to the storage cluster 20.

[0080] For example:

[0081] 1. An IO optimization method is to use a strategy generator to generate an IO optimization strategy.

[0082] Specifically, the policy generator may include one or more IO optimization algorithms. In one example, the input data of the IO optimization algorithm includes attribute information of the storage cluster 20 and / or the total amount of data of the N first data to be written. The optimization goal of the IO optimization algorithm is to improve the storage performance of the storage performance 20. Accordingly, the optimization target parameters of the IO optimization algorithm may include, but are not limited to, one or more parameters such as effective bandwidth, throughput, input / output times per second (IOPS), etc. Optionally, the IO optimization algorithm may also include constraints, which may include a write granularity range and / or a write file number range corresponding to the target performance range of the storage cluster 20.

[0083] In an optional example, attribute information of the storage cluster 20 and / or the total amount of data of N first data to be written are input into the IO optimization algorithm to obtain a first IO optimization strategy generated by the IO optimization algorithm. The first IO optimization strategy includes a first granularity and / or the number of write files for accessing data of the storage cluster 20. In an example, the number of write files can be determined based on the first granularity, or the first granularity can be determined based on the number of write files.

[0084] For example, taking a write operation as an example, assuming that the write granularity range corresponding to the target performance range of the storage cluster 20 is determined to be aKB to bKB based on the attribute information of the storage cluster 20, and the number of write files is in the range of c to d. Based on the total amount of the N first data to be written and the IO optimization algorithm, the value of the first granularity is determined to be between aKB and bKB, and M of the M second data to be written divided by the first granularity from the N first data to be written is in the range of c to d. If the value of M determined based on the initial first granularity is not between c and d, the value of the first granularity is adjusted until M is in the range of c to d.

[0085] In another embodiment, the IO optimization strategy may be determined based on one or more information such as attribute information of the storage cluster 20 , attribute information of the computing cluster 10 , and attribute information of the application used to execute the job 136 .

[0086] Among them, the attribute information of the computing cluster 10 includes but is not limited to one or more of the following: network topology information of multiple computing nodes used to execute job 136, the number of processes used to execute tasks on each computing node, and the number of computing resources (CPU, memory, etc.) on the computing node. The network topology information is used to indicate the number of the multiple computing nodes, as well as the connection relationship between computing nodes, between computing nodes and network devices such as switches, the grouping of computing nodes, whether the aggregation process spans computing nodes or spans computing node groups, etc.

[0087] In an optional example, the attribute information of the computing cluster 10 is input into an IO optimization algorithm to obtain a second IO optimization strategy generated by the IO optimization algorithm, and the second IO optimization strategy includes aggregation information, and the aggregation information is used to indicate the aggregation method. Exemplarily, the aggregation information includes but is not limited to one or more information such as the number of aggregation groups, the number of processes included in each aggregation group, the identifier of the aggregation process, and the identifier of each process in the aggregation group (used to indicate which processes' data are aggregated to which aggregation process). For example, if a type of aggregation information includes the number of processes included in each aggregation group is 5, it indicates that N processes are grouped in units of 5, and each aggregation group includes 5 processes. The 5 processes can be located in the same computing node, or the computing nodes where the 5 processes are located are connected to the same switch, so as to avoid transmitting the data to be aggregated across nodes, groups, or multiple layers of networks, thereby improving data aggregation performance.

[0088] The attribute information of the application may include but is not limited to one or more of the following: the number of processes included in the application, the IO mode of each process, whether the actions between parallel processes are the same, the total amount of data generated in each round of iteration, the total amount of data generated by a process in each round of iteration, whether the data (data format, data size, paradigm) changes between each round of iteration, etc.

[0089] In one optional example, application attribute information is input into an IO optimization algorithm to obtain a third IO optimization strategy generated by the IO optimization algorithm. The third IO optimization strategy can be used to indicate one or more of the following: which data to aggregate, the optimized IO pattern, whether the IO optimization strategy for each iteration is the same, etc. For example, if the actions of each iteration process are the same, the IO optimization strategy for each iteration can be the same, and the IO optimization strategy for each subsequent iteration can continue to use one or more IO optimization strategies determined in the first iteration.

[0090] Of course, the above information can also be combined and input into the IO optimization algorithm. For example, the input data of one IO optimization algorithm includes the attribute information of the storage cluster 20 and the attribute information of the computing cluster 10, while the input data of another IO optimization algorithm includes the attribute information of the storage cluster 20, the attribute information of the computing cluster 10, and the attribute information of the application. In addition, the input data of the IO optimization algorithm can also include other data. Any IO optimization algorithm that improves the storage performance of the storage cluster is applicable to the embodiments of the present application.

[0091] Second, another IO optimization method in the policy manager is to determine the IO optimization policy based on the configuration information submitted by the user. The configuration information may include one or more IO optimization policies, that is, the IO optimization policy can be specified by the user, which provides users with flexibility in configuring IO optimization policies.

[0092] In summary, the policy manager can generate one or more IO optimization policies for the storage cluster 20. As shown in FIG8 , the policy manager can send the one or more IO optimization policies to the process manager, which then reorganizes the N first data to be written according to the one or more IO optimization policies.

[0093] It should be noted that (1) the IO optimization strategies listed above are only examples. In practice, each IO optimization strategy may also include other information beyond the above examples, and the IO optimization strategies applicable to this embodiment are not limited to the several IO optimization strategies listed above. In addition, the above IO optimization algorithm is also an example, and this embodiment does not limit this. Any IO optimization method that can improve the performance of the storage cluster is applicable to the embodiment of this application. (2) There is no strict time sequence limit between step 702 and step 701. Step 702 can be executed before, after, or at the same time as step 701. That is, the first granularity can be determined before obtaining the N first data to be written, or after obtaining the N first data to be written, or both. (3) The attribute information of the storage cluster, the attributes of the computing cluster, and the attribute information of the application introduced above are only examples. In an optional manner, the attribute information of the storage cluster 20 can also directly include the value of the first granularity. In this case, it is not necessary to determine the first granularity based on the IO optimization algorithm.

[0094] Step 703 : Adjust the N first data to be written into M second data to be written according to the first granularity.

[0095] Taking the first granularity as the write granularity as an example, N first data to be written are adjusted to M second data to be written according to the first granularity, wherein the size of each second data to be written is the first granularity. Step 703 includes dividing the first data to be written that is larger than the first granularity among the N first data to be written according to the first granularity; and / or splicing multiple first data to be written that are smaller than the first granularity among the N first data to be written into the first granularity. For example, the first granularity is 8KB, and assuming that the size of each first data to be written is 4KB, then every two first data to be written among the N first data to be written are spliced ​​into one 8KB second data to be written. For another example, if the size of a certain first data to be written is 16KB, then the first data to be written is divided into two 8KB second data to be written.

[0096] The above operations can be performed by an IO module. The IO module provided in this embodiment also has the function of optimizing the IO mode of an application without modifying the application's program code. For example, if an application uses a first IO mode to read and write data, the IO module can use a second IO mode to read and write the data after obtaining the application's data to be read or written. The first IO mode and the second IO mode can be different. Of course, if the storage performance of the first IO mode is higher, that is, the first IO mode does not need to be optimized, the second IO mode may also be the same as the first IO mode.

[0097] Exemplarily, the difference between the second IO mode and the first IO mode (or understood as the difference before and after IO optimization) may include one or more of the following: a different number of write IOs, a different write granularity, a different number of written files, a different format of the written files, etc. As shown in Figure 8, the first IO mode writes one file for each process, and N processes write N files. The second IO mode reorganizes the N to-be-written data of the N processes into M to-be-written data, and writes the M to-be-written data in parallel to M files. If M is less than N, then there is no need to write N files at this time (hence the dashed line in Figure 8). That is, the application's IO call originally intended to write N files to the storage cluster 20. After data reorganization, the number of files written to the storage cluster 20 by the IO module may be reduced.

[0098] For example, Figure 10 illustrates a mapping relationship between a POSIX file and the data format of an I / O module. As shown in Figure 10 , assume that the application includes N processes, the application runs for K iterations, and each process writes X files per iteration. In conjunction with Figure 11(a), assuming the application uses rank 0 I / O mode, Figure 11(a) illustrates the data read and write operations during one iteration of the application. Taking the first iteration as an example, as shown in Figure 11 , the N processes of the application write data block 1 from the first iteration to file 1, data block 2 from the first iteration to file 2, and so on, until data block X from the first iteration is written to file X. In other words, each process writes X files in the first iteration. The I / O operations in each subsequent iteration follow the same pattern as the first iteration, meaning each process writes X files per iteration, but the files written may vary from one iteration to another. The total number of files written in the K iterations of the application can be N × X × K.

[0099] Assuming that the number of processes on each computing node is Q, an IO optimization strategy includes: taking each computing node as an object, treating the Q processes on a computing node as an aggregation group, one aggregation group corresponding to one file, adjusting the multiple first data to be written generated by the 1st to Kth rounds of iteration of the Q processes in an aggregation group into one or more second data to be written, and writing the one or more second data to be written to the file corresponding to the aggregation group. Based on this IO optimization strategy, the total number of write files involved in the original IO operation of the application, N×K×X, can be reduced to P=N / Q. In conjunction with Figure 11 (b), assuming that the IO mode indicated by the IO optimization strategy is the aggregation mode, Figure 11 (b) exemplarily shows the data read and write operations of a round of iterative process based on the IO optimization strategy. Still taking the first round of iteration as an example, in a possible implementation, the aggregation process in aggregation group 1 adjusts all data blocks (data block 1 to data block X) generated by process 1 to process Q in the first round of iteration into at least one second data block, and writes the at least one second data block to file 1. The aggregation processes in aggregation group 2 adjust all data blocks generated by processes Q+1 through 2Q in the first iteration into at least one second data block and write this at least one second data block to file 2. Similarly, the aggregation processes in aggregation group P adjust all data blocks generated by processes NQ through N in the first iteration into at least one second data block and write this at least one second data block to file P. The IO operations in each subsequent iteration are identical to those in the first iteration, meaning that each process writes P files in each iteration, and each iteration writes the same P files.

[0100] The above-mentioned IO optimization strategy may also specifically include a data aggregation method. For example, a data aggregation method for implementing (b) of Figure 11 includes intra-process aggregation and inter-process aggregation. For example, in combination with Figure 10, taking the first round of iteration as an example, process 1 first aggregates the data blocks 1 to data blocks X generated in the first round of iteration into a first data block within the process, and the remaining processes in the aggregation group 1 perform the same operation with reference to process 1. Then, except for the aggregation process in the aggregation group 1, each process sends its own aggregated first data block to the aggregation process, and the aggregation process further performs inter-process aggregation on the multiple first data blocks received into at least one second data block, and the aggregation process writes the at least one second data block into file 1.

[0101] In conjunction with Figure 9, the functional modules of the communication layer in the IO module can be used to exchange data between the IO modules of the process to perform intra-process data aggregation and inter-process data aggregation. For example, the IO module in the process within the aggregation group can send the first data block and the metadata of the first data block (which may include the metadata of each data block in the first data block) to the aggregation process through the MPI plug-in. If the processes in the aggregation group are located on different computing nodes, data can be transferred across computing nodes through the remote direct memory access (RDMA) plug-in. The process manager in the aggregation process instructs the serializer to reorganize multiple first data blocks into at least one second data block based on the IO optimization strategy. The serializer is used to record the order of aggregated or split data, etc. For this part, please refer to the existing functional introduction of MPI, RDMA, and serializer, which will not be repeated here.

[0102] It is worth noting that the above-mentioned IO optimization strategy can be generated by an IO optimization algorithm or configured by the user through configuration information. In this method, since most HPC applications have basically the same IO paradigm under the same calculation example and scale, end-to-end closed-loop write-read mapping can be achieved with zero code changes by loading simple configuration information.

[0103] Step 704 : Write the M second data to be written into the storage cluster 20 .

[0104] In one embodiment, M second data to be written are written in parallel into M files of the storage system 20 .

[0105] Continuing to refer to FIG9 , the transport layer of the IO module includes a file system private interface exporter and a POSIX exporter, etc. In step 704, when the IO module writes the second data to be written to the storage cluster 20, it can call one of the interfaces to write the data to the storage cluster 20. At this time, the file format of the written data will be the file format corresponding to the IO interface actually called by the IO module. For example, in step 701, the IO module intercepts the application's call to the HDF5 interface. In the prior art, the IO operation that calls the HDF5 interface will generate a file in the HDF5 format to save the data to be written by the call. In this application, when the IO module writes the data to be written to the storage cluster 20, it may call the file system private interface exporter. In this case, a file in a private format corresponding to the file system private interface will be generated. In other words, the IO operation optimized by the IO optimization strategy may also change the file format written by the application.

[0106] Through the above design, without changing the program code of the application, the IO interface commonly used in large-scale high-performance applications, such as the transparent bridge IO interface, can be used to plan an IO optimization strategy based on the deployment characteristics of the storage cluster, so that the application can achieve performance improvement without changing the program code. Since there is no need to change the program code, there is no development cost. In addition, the application can dynamically perceive the properties of the underlying storage cluster, and optimize the IO size, number of files to be written, and file format based on the properties of the storage cluster, completely decoupling the application logic from the underlying IO behavior, and controlling the adjusted IO behavior within a performance range that is better for the back-end storage cluster, thereby improving the effective IO bandwidth of the back-end storage cluster, making full use of the bandwidth of the storage cluster, and significantly improving the actual IO performance of the application.

[0107] In addition to the functions described above, the IO module provided in this embodiment can also be used to convert file formats. In this embodiment, the IO module can include multiple file format converters, such as the NetCDF format converter and HDF5 format converter shown in Figure 9. The NetCDF format converter can be used to convert files from proprietary formats to NetCDF format, or from other formats such as HDF5 to NetCDF format. The HDF5 format converter can be used to convert files from proprietary formats to HDF5 format, or from other formats such as NetCDF to HDF5 format.

[0108] When the IO module calls the POSIX exporter to write data to storage cluster 20, the data is saved as a POSIX file in storage cluster 20. The POSIX file extractor can be used to extract POSIX files from storage cluster 20. Optionally, the IO module can convert POSIX files to other formats using a format converter.

[0109] Table 1 shows the characteristics of several traditional IO interfaces.

[0110] Table 1

[0111] As shown in Table 1, among the various data formats, formats with good post-processing compatibility generally have poor performance in large-scale parallel execution. Data formats with good performance generally have poor compatibility with third-party post-processing tools.

[0112] The IO module provided by the embodiments of the present application can transparently bridge IO interfaces with good compatibility but poor performance in large-scale parallel computing, providing performance equivalent to or even higher than that of IO interfaces with good performance but poor compatibility. For example, bridging applications to IO interfaces such as PNetCDF, NetCDF, and HDF5, optimizing the IO operations of applications on these IO interfaces through the IO module, and still supporting applications or third parties to access application-generated data in PNetCDF, NetCDF, and HDF5 formats. This improves the performance of the original IO interface while retaining the good compatibility of the interface.

[0113] The functional modules included in the offline layer in the IO module introduced above can be deployed on the computing node, which can be within or outside the process of the computing node. Alternatively, the IO module does not include the functional modules of the offline layer, and these functional modules are deployed on the storage node, without specific limitation.

[0114] Optionally, after the IO module writes the data to the storage cluster 20, the storage cluster 20 can also perform offline data reorganization, such as reorganizing one or more files in a private format into one file. The format of this file can be a private format or other formats. In this way, when an application wants to read multiple files in a private format, it can directly return the reorganized file to improve the application's reading performance.

[0115] This application provides IO optimization methods for some specific applications, which are introduced below in conjunction with Table 2. Table 2 shows an example of IO methods commonly used in typical applications in the field of particle physics simulation.

[0116] Table 2

[0117] Taking simulation as an example, in the field of particle physics simulation, applications often need to frequently read and write a large amount of small blocks of data to store the simulation results of particles, such as the position, velocity and other status data of the particles. Such applications also have relatively high requirements for the frequency of IO operations, and often need to frequently record the status of particles during the simulation process. When applications use traditional data formats with better compatibility such as PHDF5 and PNetCDF to read and write such data, they will encounter serious performance bottlenecks. If a private format with relatively good performance is used, it will not be compatible with third-party post-processing tools. When using high-performance IO libraries such as ADIOS2 to read and write such data, since the application needs to write small blocks of data frequently, and each write requires the generation of a fixed-size metadata block, it will lead to the problem of too large a proportion of metadata.

[0118] To address this issue, embodiments of the present application provide a method that allows users to indicate, through configuration information, that data does not require order preservation, i.e., out-of-order data. For example, the configuration information may indicate that the simulation results of a particle are data that does not require order preservation. Accordingly, the IO module may aggregate multiple received small blocks of data including the particle simulation results into a second data to be written based on the configuration information. For this second data to be written, the metadata of the multiple small blocks of data in the second data to be written may not be recorded, thereby reducing metadata overhead.

[0119] Through the above design, the IO module provided in this embodiment can achieve performance similar to that of high-performance IO libraries such as ADIOS2 through performance optimization. At the same time, through methods such as data block aggregation and reorganization, and offline data reorganization in the storage cluster, the metadata ratio is reduced to a level similar to that of traditional IO libraries such as PHDF5 and PNetCDF. Furthermore, by transparently bridging with highly compatible general interfaces such as PHDF5 and PNetCDF, consistent third-party tool compatibility is achieved.

[0120] The above describes the data processing method for write operations. When performing read operations, the granularity of the read data (referred to as the second granularity) can be determined. Data is read from the storage cluster 20 based on the second granularity. The second granularity can be determined by the aforementioned IO optimization strategy and will not be further described here.

[0121] It should be noted that (1) the system shown in FIG1 is only an example, and the system to which this embodiment applies may have more or fewer devices relative to those shown in FIG1. ​​(2) The structure of the IO module shown in FIG9 is only an example, and in actual applications, the IO module may have more or fewer functional modules or layer structures. In addition, the IO module may also have other names. For example, logically, the IO module is located between the application process and the storage cluster, and is used to connect the application process and the storage cluster. Therefore, in some embodiments, the IO module may also be called IO middleware. (3) The layer structure and layer names included in the IO module shown in FIG9 and the names of the functional modules included in each layer are only examples, and may have different names in different records. This embodiment does not limit this.

[0122] The method provided by the embodiment of the present application is described above in conjunction with the accompanying drawings, and the device provided by the embodiment of the present application is described below in conjunction with the accompanying drawings.

[0123] Based on the same inventive concept as the method embodiment, the present application also provides a data processing device for executing the method of the method embodiment of FIG7 . As shown in FIG12 , device 1200 includes an acquisition module 1201 , a processing module 1202 , and an execution module 1203 , and optionally, an adjustment module 1204 and a determination module 1205 . Specifically, in device 1200 , each module is connected via a communication path.

[0124] The acquisition module 1201 is used to acquire N first data to be written of the multiple processes; for details, please refer to the description of step 701 in the method embodiment of Figure 7, which will not be repeated here.

[0125] Processing module 1202 is used to adjust the N first data to be written into M second data to be written according to the first granularity of the data accessed to the storage cluster, where the size of each second data to be written is the first granularity, and the first granularity is determined according to the attributes of the storage cluster; for details, please refer to the description of step 703 in the method embodiment of Figure 7, which will not be repeated here.

[0126] The execution module 1203 is configured to write the M second data to be written into the storage cluster. For details, please refer to the description of step 704 in the method embodiment of FIG7 , which will not be repeated here.

[0127] In one possible implementation, when the processing module 1202 adjusts the N first data to be written into M second data to be written according to the first granularity of the data accessing the storage cluster, it is specifically used to: split the first data to be written that is larger than the first granularity among the N first data to be written according to the first granularity; and / or splice the first data to be written that is smaller than the first granularity among the N first data to be written into the first granularity.

[0128] In a possible implementation, the adjustment module 1204 is configured to adjust the size of the first granularity when M is greater than a preset value, until M is less than or equal to the preset value.

[0129] In one possible implementation, the processing module 1202 is further used to determine multiple first data to be read of the multiple processes; the execution module 1203 is further used to obtain at least one second data to be read from the storage cluster according to the second granularity of accessing the data of the storage cluster, and the at least one second data to be read includes the multiple first data to be read, and the size of each second data to be read is the second granularity, and the second granularity is determined according to the attributes of the storage cluster.

[0130] This application also provides a computing device 1300. As shown in Figure 13, computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. Processor 1304, memory 1306, and communication interface 1308 communicate with each other via bus 1302. Computing device 1300 can be a server, a storage array, a hard disk enclosure, etc. It should be understood that this application does not limit the number of processors and memories in computing device 1300.

[0131] Bus 1302 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG13 shows a single line, but this does not imply a single bus or type of bus. Bus 1302 may include a path for transmitting information between various components of computing device 1300 (e.g., memory 1306, processor 1304, and communication interface 1308).

[0132] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0133] The memory 1306 may include volatile memory, such as random access memory (RAM). The processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), etc., without limitation.

[0134] In one embodiment, the computing device 1300 is configured to execute the functions of the data processing apparatus 1200 . Memory 1306 stores executable program code, and processor 1304 executes the executable program code to implement the functions of the aforementioned acquisition module 1201 , processing module 1202 , execution module 1203 , adjustment module 1204 , and determination module 1205 , thereby implementing the data processing method. Specifically, memory 1306 stores instructions for the computing device 1300 to execute the data processing method provided herein.

[0135] The communication interface 1308 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1300 and other devices or a communication network.

[0136] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0137] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0138] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0139] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0140] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0141] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0142] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0143] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0144] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0145] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0147] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0149] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applied to a computing cluster, the computing cluster includes a plurality of computing nodes, the computing cluster is connected to a storage cluster, the storage cluster is used to store data generated by the computing cluster, a plurality of application processes are run in the plurality of computing nodes, each process executes a portion of the data of the application, the method includes: Obtaining N first to-be-written data of the multiple processes; adjusting the N first to-be-written data into M second to-be-written data according to a first granularity of data accessing the storage cluster, wherein a size of each of the second to-be-written data is the first granularity, and the first granularity is determined according to an attribute of the storage cluster; and The M second data to be written are written into the storage cluster.

2. The method according to claim 1, characterized in that Adjusting the N first to-be-written data into M second to-be-written data according to a first granularity of accessing data of the storage cluster includes: The first data to be written that are larger than the first granularity among the N first data to be written are divided according to the first granularity; and / or the first data to be written that are smaller than the first granularity among the N first data to be written are spliced ​​into the first granularity.

3. The method according to claim 1 or 2, characterized in that The method further comprises: When the M is greater than a preset value, the size of the first granularity is adjusted until the M is less than or equal to the preset value.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determine a plurality of first to-be-read data of the plurality of processes; At least one second data to be read is obtained from the storage cluster according to a second granularity of accessing the data of the storage cluster, wherein the at least one second data to be read includes the multiple first data to be read, and the size of each second data to be read is the second granularity, and the second granularity is determined according to the attributes of the storage cluster.

5. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring N first to-be-written data of the multiple processes; a processing module, configured to adjust the N first to-be-written data into M second to-be-written data according to a first granularity of data accessing the storage cluster, wherein a size of each of the second to-be-written data is the first granularity, and the first granularity is determined according to an attribute of the storage cluster; and An execution module is used to write the M second data to be written into the storage cluster.

6. The device according to claim 5, characterized in that When the processing module adjusts the N first data to be written into M second data to be written according to the first granularity of the data accessing the storage cluster, it is specifically used to: split the first data to be written that is larger than the first granularity among the N first data to be written according to the first granularity; and / or splice the first data to be written that is smaller than the first granularity among the N first data to be written into the first granularity.

7. The device according to claim 5 or 6, characterized in that The device also includes an adjustment module; the adjustment module is used to adjust the size of the first granularity when M is greater than a preset value, until M is less than or equal to the preset value.

8. The device according to any one of claims 5 to 7, characterized in that: The processing module is also used to determine multiple first data to be read of the multiple processes; the execution module is also used to obtain at least one second data to be read from the storage cluster according to the second granularity of accessing the data of the storage cluster, and the at least one second data to be read includes the multiple first data to be read, and the size of each second data to be read is the second granularity, and the second granularity is determined according to the attributes of the storage cluster.

9. A computing device, characterized in that The device comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the device executes the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or an instruction. When the computer program or the instruction is executed by the communication device, the method according to any one of claims 1 to 4 is implemented.

Images