Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method and related equipment. The method is applied to a computing node, a parallel file system client process and an application process run in the computing node, the application process comprises a first bridging interface, the parallel file system client process comprises a second bridging interface, and the first bridging interface is connected with the second bridging interface. In the application, in response to a read-write instruction, a computing node can transmit a target read-write request to a second bridging interface of a parallel file system client process through a first bridging interface of an application process, and the read-write instruction is used for requesting target data; the target read-write request is used for requesting the target data from the parallel file system client process. Next, in response to the target read-write request, the computing node may send a file request to the storage system through the parallel file system client process, the file request being used to request the storage system for the target data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and related equipment. Background Art

[0002] With the rapid development of high-performance computing (HPC) application scenarios (such as meteorology, education and research, manufacturing, electronic design automation (EDA), and life sciences), input / output (IO) has become a major factor affecting HPC. Improving IO performance has become a key point in HPC performance optimization.

[0003] The computing system includes computing nodes and a storage system. The computing nodes have built-in software entities including application processes and a kernel. To achieve data parallelism, the computing nodes can also have built-in parallel file system client processes, which are used to access data from the storage system serving as a parallel file system. The application process can send read and write requests to the kernel. The kernel calls the parallel file system interface through the virtual file system (VFS) and sends the read and write requests to the parallel file system client process. The parallel file system client process converts the read and write requests into file requests and sends the file requests to the storage system via remote direct memory access (RDMA). The need to go through steps such as the VFS in the kernel requires a large overhead, which becomes a bottleneck for further improving IO performance. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method and related devices for allowing an application process to bypass the kernel and send read and write requests to a parallel file system client, thereby avoiding kernel overhead and improving IO performance.

[0005] The first aspect of the present application provides a data processing method, which is applied to a computing node, in which a parallel file system client process and an application process are running, the application process includes a first bridge interface, the parallel file system client process includes a second bridge interface, and the first bridge interface is connected to the second bridge interface. In the present application, in response to a read / write instruction, the computing node can transmit a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process. Then, in response to the target read / write request, the computing node can send a file request to the storage system through the parallel file system client process, and the file request is used to request the target data from the storage system. Then, since the application process can be directly connected to the parallel file system client process, thereby bypassing the kernel, the kernel overhead is avoided and the IO performance is improved.

[0006] In some possible implementations, sending the file request to the storage system through the parallel file system client process includes transmitting the file request to the storage system through Remote Direct Data Access (RDMA) of the parallel file system client process. RDMA is used to transfer data directly to a storage area of ​​a computing node over a network, rapidly moving data from one system to a memory of a remote storage system without impacting the operating system. This eliminates the need for processing capabilities of the computing node, eliminates external memory copying and file exchange operations, and frees up bus space and computing resources to improve application system performance.

[0007] In some possible implementations, in response to a file receiving instruction, the parallel file system client process receives the target file sent by the storage system and passes the target file to the application process. The target file includes the target data, so that the computing node can obtain the required target data.

[0008] In some possible implementations, the application process includes a glibc interface, through which the first bridge interface can be called, so that the application process executes the step of transmitting the target read and write request to the second bridge interface of the parallel file system client process through the first bridge interface.

[0009] In some possible implementations, the parallel file system client process includes a first cache, the target read / write request includes multiple read / write requests, metadata of each of the target read / write requests is stored in the first cache to obtain target metadata of the target data, a file request is generated based on the target metadata, the file request includes the target metadata, and the file request is sent to the storage system via the parallel file system client process. Treating each file request as one I / O operation reduces I / O times and improves I / O performance of the computing system.

[0010] In some possible implementations, the application process includes a second cache. Before sending a file request to the storage system through the parallel file system client process, the application process stores metadata of each of the multiple read and write requests in the second cache to obtain target metadata of the target data, and transmits the target read and write request to the parallel file system client process through the application process. The target read and write request is a read and write request, and the target read and write request includes the target metadata. The target read and write request is treated as an IO, thereby reducing the number of IO times and improving the IO performance of the computing system.

[0011] In some possible implementations, the application process includes a first locking interface, and the parallel file system client process includes a second locking interface. Therefore, before the target read / write request is transmitted to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, a locking instruction is transmitted to the second lock interface of the parallel file system client process through the first lock interface of the application process. The locking instruction is used to prohibit processes other than the application process from accessing the target data. This allows the application process to only access the data indicated by the read / write request when it sends a read / write request to the parallel file system client process, thereby preventing other application processes from accessing the data indicated by the read / write request and achieving strong semantic consistency among multiple application processes for the same piece of data.

[0012] In some possible implementations, the computing node includes a peripheral data processing unit (DPU) configured to run the parallel file system client process, wherein the second bridge interface is a high-speed serial computer interconnect express (PCIe) bus standard. By using the parallel file system client process in the peripheral DPU, steps such as avoiding the kernel's use of the VFS for read and write requests are implemented, further improving the computing system's IO performance.

[0013] A second aspect of the present application provides a computing node, which is used to execute any one of the methods described in the first aspect.

[0014] The third aspect of the present application provides a computing system, comprising: a computing device and a storage system, wherein:

[0015] The computing node is used to execute any one of the methods executed by the computing node in the first aspect above;

[0016] The storage system is used to receive a file request sent by the computing node, and send a target file to the computing node based on the file request. The file request is used to request the target data from the storage system, and the target file includes the target data.

[0017] A fourth aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided by the first aspect or any possible implementation of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product, which includes computer-executable instructions, which are stored in a computer-readable storage medium; at least one processor of a device can read the computer-executable instructions from the computer-readable storage medium, and at least one processor executes the computer-executable instructions so that the device implements the method provided by the above-mentioned first aspect or any possible implementation of the first aspect.

[0019] In a sixth aspect, the present application provides a communication device, which may include at least one processor, a memory, and a communication interface. The at least one processor is coupled to the memory and the communication interface. The memory is configured to store instructions, the at least one processor is configured to execute the instructions, and the communication interface is configured to communicate with other communication devices under the control of the at least one processor. When executed by the at least one processor, the instructions cause the at least one processor to perform the method of the first aspect or any possible implementation of the first aspect.

[0020] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0021] In a possible design, the chip system may further include a memory for storing necessary program instructions and data. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0022] Among them, the technical effects brought about by the second to seventh aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1-1A schematic diagram of the structure of a computing system provided in an embodiment of the present application;

[0024] Figure 1-2 A schematic diagram of the hardware structure of a computing node provided in an embodiment of the present application;

[0025] Figure 1-3 A schematic diagram of the interaction between various software entities of a computing node in an embodiment of the present application;

[0026] Figure 2-1 Another schematic diagram of the interaction between various software entities of a computing node in an embodiment of the present application;

[0027] Figure 2-2 This is a schematic diagram of an embodiment of a data processing method described in an embodiment of the present application;

[0028] Figure 3-1 Another schematic diagram of the interaction between various software entities of a computing node in an embodiment of the present application;

[0029] Figure 3-2 This is a schematic diagram of an embodiment of a data processing method described in an embodiment of the present application;

[0030] Figure 4 Another schematic diagram of the interaction between various software entities of a computing node in an embodiment of the present application;

[0031] Figure 5 Another schematic diagram of the interaction between various software entities of a computing node in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a computing node provided in an embodiment of the present application;

[0033] Figure 7 A schematic structural diagram of a communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application provide a data processing method and related devices for allowing an application process to bypass the kernel and send read and write requests to a parallel file system client, thereby avoiding kernel overhead and improving IO performance.

[0035] The embodiments of the present application are described below with reference to the accompanying drawings.

[0036] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0037] See also Figure 1-1 , the present application can be applied to a computing system 100, which includes a storage system 110 and a computing node 120. In some possible implementations, the storage system 110 and the computing node 120 can be connected via a communication network or directly via a data transmission medium, which is not limited here.

[0038] In some possible implementations, the storage system 110 may be a single server or a distributed storage system composed of multiple servers, which is not limited here.

[0039] Since servers need to respond to service requests and process them to provide reliable services, they should generally be able to undertake and guarantee services. They should also have strong processing capabilities, high stability, high reliability, high security, scalability, and manageability. In the embodiments of the present application, the server may be an x86 server, also known as a complex instruction set computer (CISC) architecture server, commonly known as a personal computer (PC) server. This server is based on the PC architecture and uses an Intel or other x86 instruction set-compatible processor chip and a Windows operating system.

[0040] Servers may vary greatly due to different configurations or performance, and may include at least one central processing unit (CPU) (e.g., at least one processor) and memory, and at least one storage medium for storing applications or data (e.g., at least one mass storage device). The memory and storage medium may be temporary storage or persistent storage. The program stored in the storage medium may include at least one module, and each module may include a series of instruction operations on the server. Furthermore, the central processing unit may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the server. The server may also include at least one power supply, at least one wired or wireless network interface, at least one input and output interface, and / or at least one operating system, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, NetWare, etc. In some feasible implementations, the server may also be a cloud server, which is not limited here.

[0041] The distributed storage system is a server cluster consisting of several servers, or a cloud computing service center, which is not limited in the embodiments of this application. The multiple servers in the distributed storage system can be connected through a data transmission medium, one or more switches, or the Internet, which is not limited here.

[0042] The computing node 120 can be a general-purpose computing device or a special-purpose computing device, including but not limited to any electronic device such as a portable computing device, a personal desktop computing device, a network server, a tablet computer, a mobile phone, a personal digital assistant (PDA), or a combination of two or more of the above. This application does not impose any limitation on the specific implementation form of the computing node 120.

[0043] For example, Figure 1-2 As shown in FIG, a schematic diagram of the hardware structure of the computing node 120 provided in an embodiment of the present application is provided. Figure 1-2 As shown, the hardware structure of the computing node 120 may include components such as a processing unit 121 , a storage unit 122 , a network interface 123 (also known as a network card or network adapter, etc.), a bus 124 and a memory unit 125 .

[0044] The processing unit 121 is the control center of the computing node 120, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. Optionally, the processing unit 121 includes one or more central processing units (CPUs), which can be single-core processors or multi-core processors, which are not limited here. The processing unit 121 can use a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the technical solutions provided in this application, which are not limited here.

[0045] The storage unit 122 may be one or more non-volatile memories, such as one or more disk storages, flash memories, or other non-volatile memories. The memory unit 125 may be a random access memory. The storage unit 122 may store program code, and the processing unit 121 may copy the program code to the memory unit 125 so that the processing unit 121 executes the program code in the memory unit 125 to execute the corresponding application process or operating system.

[0046] It should be noted that the operating system (such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS or an embedded operating system such as Vxworks) includes various software components and / or drivers for controlling and managing routine system tasks (such as memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components. Among them, the application program refers to a program written for a specific application purpose of the user, such as a text processor, spreadsheet, accounting application, browser, media player, aviation flight simulator, command line game, image editor, etc. The application program can directly complete the user's work. In other words, the operating system is used to serve the application program, and the application program is used to serve the user. When the application program is executed in the computing node 120, it is called an application process.

[0047] The network interface 123 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the computing node 120 and the storage system 110 or other devices or communication networks.

[0048] The above components are connected via bus 124. For example, processing unit 121 can be connected to components such as storage unit 122, network interface 123 (also known as network card or network adapter, etc.), bus 124 and memory unit 125 via bus 124. Bus 124 can be one or more, which is not limited here. Bus 124 includes advanced microcontroller bus architecture (AMBA) industry standard architecture (ISA) bus, micro channel architecture (MCA) bus, extended ISA bus, video electronics standard association (VESA) local bus, and peripheral component interconnect (PCI) bus, etc.

[0049] Figure 1-2 The hardware structure shown and the above description are applicable to various computing devices provided in the embodiments of the present application, and are applicable to executing various data processing methods provided in the embodiments of the present application.

[0050] Figure 1-2 The computing node 120 is only an example, and the computing node 120 may include Figure 1-2 The computing node 120 may include more or fewer components than those shown, or may have different component configurations. According to specific needs, those skilled in the art will understand that the computing node 120 may also include hardware devices that implement other additional functions. Those skilled in the art will understand that the computing node 120 may also include only the devices necessary to implement the embodiments of the present invention, and does not necessarily include the components. Figure 1-2 All devices shown in .

[0051] In an embodiment of the present application, the program code stored in the storage unit 122 can be copied to the memory unit 125. When the processing unit 121 executes the program code through the memory unit 125, a software entity running in the computing node 120 can be implemented, including a kernel and an application process.

[0052] The kernel is the most fundamental component of an operating system, its core. It isolates modules closely related to the compute node's hardware (such as interrupt handlers and device drivers), basic, common, and frequently executed modules (such as clock management and process scheduling), and critical data structures, placing them in memory and protecting them, thereby reducing the operating system's overhead. Furthermore, the kernel can be considered the first-layer software extension based on the compute node's 120 hardware, providing the operating system's most basic functions and serving as the foundation for its operation. The kernel manages the operating system's application processes, device drivers, and files, ultimately determining its performance and stability.

[0053] An application process is a software entity that is executed by the processing unit 121 based on the program code in the memory unit 125 after the corresponding program code is copied to the memory unit 125. An application is an instance of the operating system for resource allocation. From another perspective, an application process is an instance of a running application.

[0054] In some possible implementations, in order to support parallel reading and writing of data, a parallel file system client process may also be run in the computing node 120. It should be noted that the parallel file system client process is a software entity of the parallel file system running in the computing node 120. A parallel file system is a type of file system, commonly used in the field of high-performance computing. A parallel file system can better support parallel computing and large-scale data reading and writing. A file system is a method and data structure used in an operating system to define files in a storage unit, that is, a method of organizing files on a storage unit. It is the most common and simplest way to use storage. The parallel file system can run on the server and the client. When the parallel file system runs on the client's device, it is a parallel file system client process. The parallel file system client process is used to interact with the storage system so that the application process can obtain the required data from the storage system through the parallel file system client process.

[0055] like Figure 1-3FIG. 1 is a schematic diagram illustrating the interaction between various software entities in a computing node 120. An application process 120-1, a kernel 120-2, and a parallel file system client process 120-3 are running in the computing node 120. In the application process 120-1, an application can send read and write requests to the kernel 120-2 via the glibc interface. After receiving the read and write requests, the kernel 120-2 processes the read and write requests through a system call to the virtual file system (VFS), obtains a file request, and forwards the file request to the parallel file system client process 120-3 via the parallel file system interface. After receiving the file request, the parallel file system client process 120-3 can retrieve the file indicated in the file request from the storage system 110 via remote direct memory access (RDMA).

[0056] However, the read and write requests of the application process 120-1 need to pass through the kernel 120-2, and in the kernel 120-2, they need to go through steps such as VFS. Since the use of steps such as VFS in the kernel 120-2 requires a large overhead, it becomes a bottleneck for further improving IO performance.

[0057] To this end, the present application proposes a data processing method and related equipment for allowing the application process to bypass the kernel and send read and write requests to the parallel file system client to avoid kernel overhead and improve IO performance.

[0058] The present application is applied to a computing node, in which a parallel file system client process and an application process are running, the application process includes a first bridge interface, the parallel file system client process includes a second bridge interface, and the first bridge interface is connected to the second bridge interface. In the present application, in response to a read / write instruction, the computing node can transmit a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process. Then, in response to the target read / write request, the computing node can send a file request to the storage system through the parallel file system client process, and the file request is used to request the target data from the storage system. Then, since the application process can be directly connected to the parallel file system client process, thereby bypassing the kernel, the kernel overhead is avoided and the IO performance is improved.

[0059] This application provides four embodiments, namely, Embodiment 1, Embodiment 2, Embodiment 3, and Embodiment 4. Embodiment 1 introduces a technical solution in which a first bridge interface in an application process 120-1 is connected to a second bridge interface of a parallel file system client process, and the parallel file system client process has a built-in first cache; Embodiment 2 introduces a technical solution in which a second cache is built-in in the application process 120-1; Embodiment 3 introduces a technical solution in which the application process 120-1 locks data in a read / write request via a first locking interface; and Embodiment 4 introduces a technical solution in which the parallel file system client process is offloaded to a data processing unit (DPU) of an external device.

[0060] like Figure 2-1 The figure shows a schematic diagram of the interaction between the various software entities in the computing node 120 in the first embodiment of the present application. Compared with the prior art, in the first embodiment of the present application, a first bridge interface is added to the application process 120-1, and a second bridge interface is added to the parallel file system client process 120-3. In the second embodiment of the present application, the first bridge interface in the application process 120-1 can be called through the glibc interface. The application process 120-1 can send read and write requests to the parallel file system client process 120-3 through the first bridge interface. The parallel file system client process 120-3 can receive the read and write requests through the second bridge interface, convert the read and write requests into file requests, and obtain the corresponding files from the storage system 110 via RDMA. Because the read and write requests do not require the use of VFS and other steps in the kernel 120-2, the IO performance of the computing system 100 is further improved. Optionally, the parallel file system client process reduces the number of IO operations and improves IO performance by caching the target read and write requests, converting them into a file request, and sending the file request to the storage system.

[0061] The first embodiment of the present application is applied to Figure 2-1 The computing node 120 shown runs a parallel file system client process and an application process. The application process includes a first bridge interface. The parallel file system client process includes a second bridge interface and a first cache. The first bridge interface is connected to the second bridge interface.

[0062] Based on Figure 2-1 For the compute nodes shown, please refer to Figure 2-2 A data processing method according to the first embodiment of the present application mainly includes the following steps:

[0063] 201. In response to the read / write instruction, the application process transmits a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface, where the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process.

[0064] In some possible implementations, a user may operate an application process on a computing node, so that the application process receives the user's operation instructions and, based on the operation, generates read and write instructions. Alternatively, in some possible implementations, the computing node may receive read and write instructions for the application process, or the application process may generate read and write instructions while running. This is not limited here.

[0065] In an embodiment of the present application, in response to the read / write instruction, the application process may generate a corresponding target read / write request, wherein the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process.

[0066] In an embodiment of the present application, after the application process generates a target read / write request, the application process may transmit the target read / write request to the second bridge interface of the parallel file system client process. In some possible implementations, the application process may include a glibc interface, and the application process may call the first bridge interface through the glibc interface, so that the application process executes the first bridge interface to transmit the target read / write request to the second bridge interface of the parallel file system client process.

[0067] In some possible implementations, the glibc interface can simultaneously connect to the kernel and the parallel file system client process. The path through which the glibc interface connects to the kernel is called path 1, and the path through which the glibc interface connects to the first bridge interface is called path 2. In some possible implementations, while operating the application process, a user can select path 1 or path 2 for a generated target read or write request. If the user selects path 1, the target read or write request can be sent to the kernel; if the user selects path 2, the target read or write request can be sent to the parallel file system client process.

[0068] In some possible implementations, the path selected by the application process for the target read / write request can be preset. In an embodiment of the present application, the path selected by the application process for the target read / write request can be preset to path 2, that is, path 2 is the default path, which is not limited here.

[0069] In an embodiment of the present application, since the second bridge interface can be connected to the first bridge interface of the application process, the parallel file system client process can bypass the kernel and receive the target read and write requests sent by the application process through the first bridge interface through the second bridge interface, so that the application process can directly send the target read and write requests to the parallel file system client process, avoiding kernel overhead and improving IO performance.

[0070] 202. In response to the target read / write request, send a file request to the storage system through the parallel file system client process, where the file request is used to request the target data from the storage system.

[0071] In some possible implementations, after the parallel file system client process receives the target read / write request, in response to the target read / write request, the parallel file system client process may send a file request to the storage system, where the file request is used to request the target data from the storage system.

[0072] In some possible implementations, the parallel file system client process can transmit the file request to the storage system via Remote Direct Access (RDMA). RDMA is used to transfer data directly to the storage area of ​​a compute node over a network, rapidly moving data from one system to the memory of a remote storage system without any impact on the operating system. This eliminates the need for processing power on the compute node, eliminates external memory copying and file exchange operations, and frees up bus space and computing resources to improve application system performance.

[0073] In some possible implementations, such as Figure 2-1 As shown, the parallel file system client process includes a first cache, and the target read / write request includes multiple read / write requests. The parallel file system client process can then store metadata of the multiple read / write requests in the first cache to obtain target metadata for the target data. The parallel file system client process then generates a file request based on the target metadata, including the target metadata, and sends the file request to the storage system.

[0074] In some possible implementations, the first cache that can be built into the parallel file system client process can be a cache resource allocated by the computing node to the parallel file system client process. The cache resource is a storage space divided from a memory unit. For example, the size of the first cache is 16 megabytes.

[0075] In an embodiment of the present application, to reduce the number of I / O operations and further improve the I / O performance of the computing system, the parallel file system client process can first store received read and write requests in a first cache. When a certain number of read and write requests from the application process are accumulated, a file request is generated based on the multiple read and write requests received, with each file request being considered as one I / O operation, thereby reducing the number of I / O operations and improving the I / O performance of the computing system.

[0076] It should be noted that the first cache can store the metadata of each read and write request included in the target read and write request to obtain the target metadata of the target data. For example, the amount of data requested by each read and write request in the target read and write request is 1 megabyte, and the target read and write request is 16 read and write requests. Then, the generated file request is used to request 16 megabytes. In this case, only one 16-megabyte file request needs to be sent, and 16 read and write operations do not need to be sent, thereby reducing the number of IO operations and improving IO performance. It should be noted that the target read and write requests all come from the same application process and are used to request data from the same storage system.

[0077] In some possible implementations, the target read / write request may also be a read / write request, which is not limited here.

[0078] 203. In response to the file receiving instruction, receive the target file sent by the storage system through the parallel file system client process, and transfer the target file to the application process, where the target file includes the target data.

[0079] In an embodiment of the present application, after the parallel file system client process sends a file request to the storage system, the storage system can obtain the corresponding target file based on the metadata of the file request, which includes the target data. The storage system can then return the target file to the parallel file system client process, and the parallel file system client process receives a file receive instruction. In response to the file receive instruction, the parallel file system client process receives the target file sent by the storage system and delivers the target file to the application process.

[0080] In the first embodiment of the present application, the application process can send a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface, thereby bypassing the kernel, avoiding steps such as using VFS in the kernel, avoiding kernel overhead, and further improving the IO performance of the computing system. In addition, when the target read / write request includes multiple read / write requests, the parallel file system client process caches the target read / write request, converts it into a file request, and sends the file request to the storage system, thereby reducing the number of IO requests and improving IO performance.

[0081] The second embodiment of the present application is applied to Figure 3-1 The computing node 120 shown in the figure runs a parallel file system client process and an application process. The application process includes a first bridge interface and a second cache. The parallel file system client process includes a second bridge interface. The first bridge interface is connected to the second bridge interface.

[0082] Based on Figure 3-1 For the compute nodes shown, please refer to Figure 3-2 A data processing method according to the second embodiment of the present application mainly includes the following steps:

[0083] 301. In response to the read / write instruction, store metadata of the multiple read / write requests in the second cache to obtain target metadata of the target data.

[0084] In some possible implementations, a user may operate an application process on a computing node, so that the application process receives the user's operation instructions and, based on the operation, generates read and write instructions. Alternatively, in some possible implementations, the computing node may receive read and write instructions for the application process, or the application process may generate read and write instructions while running. This is not limited here.

[0085] In an embodiment of the present application, in response to the read / write instruction, the application process may generate corresponding multiple read / write requests, wherein the read / write instruction is used to request target data, and the multiple read / write requests are used to request the target data from the parallel file system client process.

[0086] 302. Deliver a target read / write request to the parallel file system client process through the application process, where the target read / write request is a read / write request and includes the target metadata.

[0087] In some possible implementations, such as Figure 3-1 As shown, the application process includes a second cache, and the target read / write request is multiple read / write requests. Therefore, the application process can store metadata of the multiple read / write requests in the second cache to obtain target metadata of the target data. Next, the parallel file system client process generates the target read / write request based on the target metadata, and the target read / write request includes the target metadata.

[0088] In some possible implementations, the second cache that can be built into the application process can be a cache resource allocated by the computing node to the application process. The cache resource is a storage space divided from the memory unit. For example, the size of the second cache is 16 megabytes.

[0089] In an embodiment of the present application, to reduce the number of I / O operations and further improve the I / O performance of the computing system, the application process can first store received read and write requests in a second cache. When a certain number of read and write requests have accumulated, a target read and write request is generated based on the multiple received read and write requests, and the target read and write request is used as a single I / O operation, thereby reducing the number of I / O operations and improving the I / O performance of the computing system.

[0090] It should be noted that the second cache can store metadata of each read / write request included in the target read / write request. For example, the amount of data requested by each of the multiple read / write requests is 1 megabyte, and there are 16 read / write requests in total. The generated target read / write request is used to request 16 megabytes. In this case, only one target read / write request of 16 megabytes needs to be sent, and 16 read / write operations do not need to be sent, thereby reducing the number of IO operations and improving IO performance. It should be noted that the multiple read / write requests of the target read / write request are all used to request data from the same storage system.

[0091] In an embodiment of the present application, after the application process generates a target read / write request, the application process may transmit the target read / write request to the second bridge interface of the parallel file system client process. In some possible implementations, the application process may include a glibc interface, and the application process may call the first bridge interface through the glibc interface, so that the application process executes the first bridge interface to transmit the target read / write request to the second bridge interface of the parallel file system client process.

[0092] In some possible implementations, the glibc interface can simultaneously connect to the kernel and the parallel file system client process. The path through which the glibc interface connects to the kernel is called path 1, and the path through which the glibc interface connects to the first bridge interface is called path 2. In some possible implementations, while operating the application process, a user can select path 1 or path 2 for a generated target read or write request. If the user selects path 1, the target read or write request can be sent to the kernel; if the user selects path 2, the target read or write request can be sent to the parallel file system client process.

[0093] In some possible implementations, the path selected by the application process for the target read / write request can be preset. In an embodiment of the present application, the path selected by the application process for the target read / write request can be preset to path 2, that is, path 2 is the default path, which is not limited here.

[0094] In an embodiment of the present application, since the second bridge interface can be connected to the first bridge interface of the application process, the parallel file system client process can bypass the kernel and receive the target read and write requests sent by the application process through the first bridge interface through the second bridge interface, so that the application process can directly send the target read and write requests to the parallel file system client process, avoiding kernel overhead and improving IO performance.

[0095] 303. In response to the target read / write request, send a file request to the storage system through the parallel file system client process, where the file request is used to request the target data from the storage system.

[0096] In some possible implementations, after the parallel file system client process receives the target read / write request, in response to the target read / write request, the parallel file system client process may send a file request to the storage system, where the file request is used to request the target data from the storage system.

[0097] In some possible implementations, the parallel file system client process can transmit the file request to the storage system via Remote Direct Access (RDMA). RDMA is used to transfer data directly to the storage area of ​​a compute node over a network, rapidly moving data from one system to the memory of a remote storage system without any impact on the operating system. This eliminates the need for processing power on the compute node, eliminates external memory copying and file exchange operations, and frees up bus space and computing resources to improve application system performance.

[0098] 304. In response to the file receiving instruction, receive the target file sent by the storage system through the parallel file system client process, and transfer the target file to the application process, where the target file includes the target data.

[0099] Please refer to step 203, which will not be described in detail here.

[0100] In the second embodiment of the present application, the application process can generate a target read and write request from the target read and write request through the built-in second cache, and send the target read and write request to the parallel file system client process through the first bridge interface, so that the parallel file system client process receives the target read and write request through the second bridge interface, bypassing the kernel, avoiding steps such as using VFS in the kernel, and further improving the IO performance of the computing system.

[0101] The third embodiment of the present application is applied to Figure 4 The computing node 120 shown in the figure runs a parallel file system client process and an application process. The application process includes a first bridge interface, and the parallel file system client process includes a second bridge interface. The first bridge interface is connected to the second bridge interface.

[0102] In the third embodiment of the present application, a first locking interface is added to the application process 120-1, and a second locking interface is added to the parallel file system client process 120-3. The first locking interface can be called through the glibc interface to send a locking instruction to the second locking interface of the parallel file system client process 120-3. Then, in the embodiment of the present application, before the target read / write request is transmitted to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, a locking instruction can be transmitted to the second locking interface of the parallel file system client process through the first locking interface of the application process. The locking instruction is used to indicate that processes other than the application process are prohibited from accessing the target data. When the application process 120-1 sends a read / write request to the parallel file system client process 120-3, only the application process 120-1 can access the data indicated by the read / write request, thereby avoiding access by other application processes to the data indicated by the read / write request, and achieving strong semantic consistency of multiple application processes for the same data.

[0103] Based on Figure 4 For the computing node shown, please refer to steps 201-203 or steps 301-304 for a data processing method provided in Example 3 of the present application, which will not be repeated here.

[0104] The fourth embodiment of the present application is applied to Figure 5 The computing node 120 shown in the figure runs a parallel file system client process and an application process. The application process includes a first bridge interface, and the parallel file system client process includes a second bridge interface. The first bridge interface is connected to the second bridge interface.

[0105] In the fourth embodiment of the present application, a peripheral data processing unit (DPU) is inserted into the computing node 120, and the parallel file system client process 120-3 can be run in the DPU. The DPU includes a first high-speed serial computer expansion bus standard (peripheral component interconnect express, PCIe), and the first PCIe can serve as a first bridge interface for the parallel file system client process 120-3. The kernel 120-2 also includes a second PCIe, and the second PCIe is used as a second bridge interface. Then, the application process 120-1 can send a read and write request to the first PCIe1 of the parallel file system client process 120-3 in the DPU through the second PCIe. The parallel file system client process 120-3 can receive the read and write request through the first PCIe, convert the read and write request into a file request, and obtain the corresponding file from the storage system 110 through RDMA. Through the parallel file system client process 120 - 3 in the peripheral DPU, it is achieved to avoid the read and write request from using VFS in the kernel 120 - 2 and other steps, thereby further improving the IO performance of the computing system 100 .

[0106] Based on Figure 5 For the computing node shown, please refer to steps 201-203 or steps 301-304 for a data processing method provided in the fourth embodiment of the present application, which will not be repeated here.

[0107] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0108] In order to better implement the above-mentioned solutions of the embodiments of the present application, relevant devices for implementing the above-mentioned solutions are also provided below.

[0109] See also Figure 6 As shown, an embodiment of the present application provides a computing node 600 , which includes a transceiver module 601 and a processing module 602 .

[0110] The processing module 602 is used to run a parallel file system client process and an application process. The application process includes a first bridge interface, and the parallel file system client process includes a second bridge interface. The first bridge interface is connected to the second bridge interface.

[0111] The processing module 602 is specifically configured to: transmit, in response to a read / write instruction, a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, wherein the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process;

[0112] The transceiver module 601 is configured to send a file request to the storage system through the parallel file system client process in response to the target read / write request, where the file request is used to request the target data from the storage system.

[0113] In some possible implementations, sending the file request to the storage system through the parallel file system client process includes:

[0114] The file request is transferred to the storage system through Remote Direct Data Access (RDMA) of the parallel file system client process.

[0115] In some possible implementations, the transceiver module 601 is further configured to:

[0116] In response to the file receiving instruction, the target file sent by the storage system is received through the parallel file system client process, and the target file including the target data is transferred to the application process.

[0117] In some possible implementations, the application process includes a glibc interface, and the processing module 602 is further configured to:

[0118] The first bridge interface is called through the glibc interface, so that the application process executes the step of transmitting the target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface.

[0119] In some possible implementations, the parallel file system client process includes a first cache, and the target read / write request includes multiple read / write requests; the processing module 602 specifically includes:

[0120] Storing metadata of each read / write request in the target read / write request in the first cache to obtain target metadata of the target data;

[0121] generating the file request based on the target metadata, wherein the file request includes the target metadata;

[0122] The transceiver module 601 is further configured to send the file request to the storage system through the parallel file system client process.

[0123] In some possible implementations, the application process includes a second cache, and the processing module 602 further includes:

[0124] Storing metadata of each of the multiple read and write requests in the second cache to obtain target metadata of the target data;

[0125] The target read / write request is transmitted to the parallel file system client process through the application process. The target read / write request is a read / write request, and the target read / write request includes the target metadata.

[0126] In some possible implementations, the application process includes a first locking interface, and the parallel file system client process includes a second locking interface;

[0127] The processing module 602 is further configured to:

[0128] A locking instruction is transmitted to the second locking interface of the parallel file system client process through the first locking interface of the application process, where the locking instruction is used to instruct that processes other than the application process are prohibited from accessing the target data.

[0129] In some possible implementations, the computing node includes a peripheral data processor DPU, which is used to run the parallel file system client process, wherein the second bridge interface is a high-speed serial computer expansion bus standard PCIe.

[0130] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and no further details will be given here.

[0131] An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a program, and the program executes some or all of the steps recorded in the above method embodiment.

[0132] Next, another communication device provided by the embodiment of the present application is introduced. Figure 7 As shown, the communication device 700 includes:

[0133] Receiver 701, transmitter 702, processor 703 and memory 704. In some embodiments of the present application, the receiver 701, transmitter 702, processor 703 and memory 704 may be connected via a bus or other means, wherein: Figure 7 The bus connection is taken as an example.

[0134] The memory 704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 703. A portion of the memory 704 may also include non-volatile random access memory (NVRAM). The memory 704 stores an operating system and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0135] Processor 703 controls the operation of communication device 700 and may also be referred to as a central processing unit (CPU). In specific applications, the various components of communication device 700 are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0136] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 703. Processor 703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 703. The above processor 703 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, and other well-known storage media in the art. The storage medium is located in the memory 704 , and the processor 703 reads the information in the memory 704 and completes the steps of the above method in combination with its hardware.

[0137] The receiver 701 can be used to receive input digital or character information and generate signal input related to relevant settings and function control. The transmitter 702 can include a display device such as a display screen. The transmitter 702 can be used to output digital or character information through an external interface.

[0138] In the embodiment of the present application, the processor 703 is used to execute the aforementioned data processing method.

[0139] In another possible design, when the computing node 700 is a chip, it includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit to enable the chip in the terminal to execute the method for sending wireless report information according to any one of the above-mentioned first aspects. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit in the terminal located outside the chip, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0140] The processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above method.

[0141] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0142] Through the description of the above embodiments, it is clear to those skilled in the art that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be implemented by means of dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0143] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0144] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: Used in a computing node, wherein a parallel file system client process and an application process are running in the computing node; the application process includes a first bridge interface, the parallel file system client process includes a second bridge interface, and the first bridge interface is connected to the second bridge interface; the method includes: In response to a read / write instruction, transmitting a target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, wherein the read / write instruction is used to request target data, and the target read / write request is used to request the target data from the parallel file system client process; In response to the target read / write request, a file request is sent to the storage system through the parallel file system client process, where the file request is used to request the target data from the storage system.

2. The method according to claim 1, characterized in that The sending of the file request to the storage system through the parallel file system client process includes: The file request is delivered to the storage system via Remote Direct Data Access (RDMA) of the parallel file system client process.

3. The method according to claim 1 or 2, characterized in that The method further comprises: In response to a file receiving instruction, the target file sent by the storage system is received by the parallel file system client process, and the target file is transferred to the application process, where the target file includes the target data.

4. The method according to any one of claims 1 to 3, characterized in that The application process includes a glibc interface, and the method further includes: The first bridge interface is called through the glibc interface, so that the application process executes the step of transmitting the target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface.

5. The method according to any one of claims 1 to 4, characterized in that The parallel file system client process includes a first cache, and the target read and write request includes multiple read and write requests; and sending the file request to the storage system through the parallel file system client process includes: Storing metadata of each read / write request in the target read / write request in the first cache to obtain target metadata of the target data; generating the file request based on the target metadata, the file request including the target metadata; The file request is sent to the storage system through the parallel file system client process.

6. The method according to any one of claims 1 to 4, characterized in that The application process includes a second cache. Before sending the file request to the storage system through the parallel file system client process, the method further includes: storing metadata of each of the plurality of read and write requests in the second cache to obtain target metadata of the target data; The target read / write request is transmitted to the parallel file system client process through the application process, where the target read / write request is a read / write request and includes the target metadata.

7. The method according to any one of claims 1 to 6, characterized in that The application process includes a first locking interface, and the parallel file system client process includes a second locking interface; Before transmitting the target read / write request to the second bridge interface of the parallel file system client process through the first bridge interface of the application process, the method further includes: A locking instruction is transmitted to the second locking interface of the parallel file system client process through the first locking interface of the application process, where the locking instruction is used to instruct that processes other than the application process are prohibited from accessing the target data.

8. The method according to any one of claims 1 to 7, characterized in that The computing node includes a peripheral data processor DPU, and the DPU is used to run the parallel file system client process, wherein the second bridge interface is a high-speed serial computer expansion bus standard PCIe.

9. A computing node, characterized in that: The computing node is configured to execute the method according to any one of claims 1 to 8.

10. A computing system, characterized in that include: Computing devices and storage systems, wherein The computing node is used to execute the method according to any one of claims 1 to 8; The storage system is used to receive a file request sent by the computing node, and send a target file to the computing node based on the file request. The file request is used to request the target data from the storage system, and the target file includes the target data.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and the program causes a computer device to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes computer-executable instructions, which are stored in a computer-readable storage medium; at least one processor of a device reads the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions so that the device performs the method according to any one of claims 1 to 8.

13. A communication device, characterized in that: The communication device includes at least one processor, a memory and a communication interface; the at least one processor coupled to the memory and the communication interface; The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other communication devices under the control of the at least one processor; When the instructions are executed by the at least one processor, the at least one processor is caused to perform the method according to any one of claims 1 to 8.

14. A chip system, characterized in that: The chip system includes a processor and a memory, the memory and the processor are interconnected via a line, instructions are stored in the memory, and the processor is used to execute the method according to any one of claims 1 to 8.

Images