Data processing method and device based on DPU, storage medium and electronic device

Abstract

The embodiment of the invention provides a DPU-based data processing method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining operator operation data of a data recombination operator and a preset data storage condition, wherein the preset data storage condition at least comprises a total storage space and an overflow coefficient of each storage device for storing the operator operation data and executing mapping write operation; based on the total storage space and the overflow coefficient of each storage device executing the mapping write operation in a preset data storage condition, calculating to obtain a storable space of each storage device executing the mapping write operation; and writing the operator operation data into each storage device executing the mapping write operation, so that the storable space of each storage device executing the mapping write operation is matched with the data volume in the operator operation data. In this way, the number of times of moving the operator operation data among the storage devices such as the DPU, the Host and the hard disk is reduced, and the data processing efficiency of the DPU is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a DPU-based data processing method and device, storage medium, and electronic device. Background Art

[0002] With the popularity of big data analysis, an advanced framework for large-scale distributed data processing, Apache Spark (i.e. Spark, a fast and universal data processing engine for large-scale data computing), has been developed. Spark can maintain the advantages of MapReduce's scalable, distributed, and fault-tolerant processing framework, while making the framework more efficient and easier to use. Spark SQL, as Spark's computing module, is specifically used to process structured data. Spark SQL allows users to use standard SQL statements to execute SQL queries and read and write, and can also use Hive SQL to execute queries and read and write to Hive warehouses.

[0003] When processing complex data calculations, since the DPU (Data Processing Unit) chip has higher performance than the CPU (Central Processing Unit), offloading SparkSQL data calculations from the CPU to the DPU can improve the performance of Spark SQL, allowing the CPU to focus on Spark's calculation scheduling and the DPU to focus on data calculations in Spark SQL.

[0004] In the related art, Spark SQL execution is offloaded to the DPU to perform calculations on data reorganization operators (such as the Shuffle operator). The data reorganization operator performs mapping and reduction processing. The result data of the mapping operation is stored on the hard disk and cannot be stored on the DPU and the Host. When the reduction operation is performed, the result data of the mapping operation stored on the hard disk will be read, which involves multiple data transfers from the hard disk to the memory. As a result, during the calculation process of the data reorganization operator, data needs to be moved multiple times between the DPU, the Host, and the hard disk, resulting in low data processing efficiency of the DPU. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a data processing method based on DPU and its equipment, storage medium and electronic device to solve or alleviate the problem in the related technology that data needs to be moved multiple times between DPU, Host and hard disk, which occupies more system resources and leads to low data processing efficiency of DPU.

[0006] According to a first aspect of an embodiment of the present application, a DPU-based data processing method is provided, the method comprising: obtaining operator operation data of a data reorganization operator and preset data storage conditions, the preset data storage conditions comprising at least the total storage space and overflow coefficient of each storage device that performs a mapping write operation to store the operator operation data; based on the total storage space and overflow coefficient of each storage device that performs a mapping write operation in the preset data storage conditions, calculating the storable space of each storage device that performs a mapping write operation; writing the operator operation data to each storage device that performs a mapping write operation, so that the storable space of each storage device that performs a mapping write operation and the amount of data in the operator operation data are adapted.

[0007] In some embodiments of the present application, the operator operation data is written to each storage device that performs a mapping write operation so that the storage space of each storage device that performs a mapping write operation and the amount of data in the operator operation data are adapted, including: determining whether the amount of data of the mapping operation result of the data reorganization operator is less than or equal to the storage space of the remote DPU that performs the mapping write operation; when the amount of data is less than or equal to the storage space of the remote DPU that performs the mapping write operation, writing the operator operation data to the remote DPU that performs the mapping write operation.

[0008] In some embodiments of the present application, the operator operation data is written to each storage device that performs a mapping write operation, so that the storage space of each storage device that performs a mapping write operation and the amount of data in the operator operation data are adapted, and also includes: when the data amount of the mapping operation result is greater than the storage space of the remote DPU that performs the mapping write operation, and is less than or equal to the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, the data amount of the mapping operation result is preferentially written to the remote DPU that performs the mapping write operation until the storage space of the remote DPU that performs the mapping write operation is full; the remaining data of the data amount of the mapping operation result is written to the remote Host that performs the mapping write operation until the storage space of the remote Host that performs the mapping write operation is full.

[0009] In some embodiments of the present application, the operator operation data is written to each storage device that performs a mapping write operation, so that the storage space of each storage device that performs the mapping write operation and the data amount in the operator operation data are adapted, and also includes: when the data amount of the mapping operation result is greater than the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, and is less than or equal to the sum of the storage spaces of each storage device that performs the mapping write operation, the data amount of the mapping operation result is preferentially written to the remote DPU that performs the mapping write operation until the storage space of the remote DPU that performs the mapping write operation is full; the remaining data of the data amount of the mapping operation result is preferentially written to the Host that performs the reduce read operation, and after the storage space of the Host that performs the reduce read operation is full, the remaining data except the data stored in the Host is written to the hard disk that performs the mapping write operation.

[0010] In some embodiments of the present application, the method further includes: the DPU performing the reduce read operation reads the operator operation data stored by the remote DPU performing the map write operation through a parallel computing communication protocol.

[0011] In some embodiments of the present application, the method further includes: the Host device that performs the reduce read operation reads the operator operation data stored in the remote Host device that performs the mapping write operation through the memory access communication protocol; and moves the operator operation data in the Host device that performs the reduce read operation to the DPU that performs the reduce read operation.

[0012] In some embodiments of the present application, the method further includes: the hard disk performing the reduce read operation reads the operator operation data stored in the hard disk performing the mapping write operation through a network transmission protocol; and the operator operation data in the hard disk performing the reduce read operation is moved to the DPU performing the reduce read operation via the Host device performing the reduce read operation.

[0013] According to a second aspect of an embodiment of the present application, a DPU-based data processing device is provided, including: an acquisition module, which acquires operator operation data of a data reorganization operator and preset data storage conditions, the preset data storage conditions at least including the total storage space and overflow coefficient of each storage device that performs a mapping write operation to store the operator operation data; a calculation module, which calculates the storable space of each storage device that performs a mapping write operation based on the total storage space and overflow coefficient of each storage device that performs a mapping write operation in the preset data storage conditions; and a writing module, which writes the operator operation data into each storage device that performs a mapping write operation, so that the storable space of each storage device that performs a mapping write operation and the amount of data in the operator operation data are adapted.

[0014] According to the third aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the DPU-based data processing method of any of the above embodiments.

[0015] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, a DPU-based data processing method as in any of the above embodiments is implemented.

[0016] According to the DPU-based data processing method and its device, storage medium and electronic device provided in the embodiments of the present application, the operator operation data of the data reorganization operator is written into the storage space of each storage device that performs the mapping write operation through the storage space of each storage device that performs the mapping write operation, so that the storage space of each storage device that performs the mapping write operation and the data amount in the operator operation data are adapted, thereby eliminating the need to move all the operator operation data to the hard disk storage through the DPU and the Host, reducing the number of times the operator operation data is moved between the DPU, the Host and the hard disk, saving the time and system resources for moving the operator operation data, and improving the data processing efficiency of the DPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 1 is a flow chart of a DPU-based data processing method according to an embodiment of the present application;

[0019] Figure 2 The data flow diagram of the operator operation of the Shuffle operator according to one embodiment of the present application is as follows;

[0020] Figure 3 is a structural block diagram of a DPU-based data processing device according to an embodiment of the present application;

[0021] Figure 4 The figure is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0023] Specifically, Apache Spark (i.e. Spark), as a fast general-purpose computing engine for large-scale data computing, is an open source cluster computing environment similar to Hadoop (a distributed infrastructure). There are some differences between the two. Spark enables in-memory distributed data sets, can provide interactive queries, and can also optimize iterative workloads, making Spark superior in some workloads.

[0024] The DPU chip based on KPU (Kernel Processing Unit) architecture has higher performance than CPU (. Therefore, offloading the data calculation of Spark SQL from CPU to DPU can improve the performance of SparkSQL, allowing CPU to focus on Spark operation scheduling and DPU to focus on data calculation in Spark SQL. Spark SQL is a module for processing structured data based on Spark, which supports query and loading of multiple data sources, is compatible with Hive, and can use JDBC / ODBC connection method to execute SQL statements. It provides important technical support for Spark framework in structured data analysis.

[0025] When a Spark task is submitted, the DPU-based Spark acceleration solution will check whether there are operators that can be offloaded to the DPU for calculation. If so, it will be converted into a DPU execution plan. During the execution process, the operator offloaded to the DPU for calculation will process the data on the DPU. The function of data reorganization operators such as the Shuffle operator is to redistribute data according to the rules specified by SQL to ensure that the same key data arrives at the same node to improve the accuracy of data calculation. Therefore, the process of executing the data reorganization operator involves multiple data movement and data transmission. If it cannot be well optimized, the overall execution efficiency will be greatly reduced.

[0026] Data reorganization operators can be roughly divided into two parts: map operation and reduce operation. In the map operation, the data is prepared and divided in the Map stage (the shuffle process on the Map side is to partition, sort, and split the results of the Map, and then merge the data outputs belonging to the same partition together and write them on the disk, and finally get a partitioned and ordered file). In the reduce operation, the data is pulled in the Reduce stage (on the Reduce side, shuffle is mainly divided into two stages: copying Map output and sorting and merging) in order to perform subsequent calculation logic. The Shuffle on the Map side is usually called a map write operation, and the Shuffle on the Reduce side is usually called a reduce read operation.

[0027] In the related art, the DPU-based Spark acceleration solution offloads the data reorganization operator to the DPU for processing, but the result data of the mapping write operation of the data reorganization operator can only be stored on the hard disk, and cannot be stored on the DPU and the Host. In other words, the operator operation data of the mapping write operation will be moved from the DPU to the hard disk via the memory multiple times, and the reduction read operation will also move the operator operation data from the hard disk to the DPU via the memory multiple times. Therefore, the multiple movement of the operator operation data between the hard disk and the memory when Spark SQL executes the data reorganization operator becomes a bottleneck for the DPU data processing performance, occupies more system resources, and leads to the problem of low data processing efficiency of the DPU.

[0028] To solve the above problems, the present application provides a data processing method based on DPU and its equipment, storage medium and electronic device, which uses the available storage space of each storage device that performs mapping write operations to write the operator operation data of the data reorganization operator into each storage device that performs mapping write operations, so that the available storage space of each storage device that performs mapping write operations and the data amount in the operator operation data are adapted, thereby eliminating the need to move all the operator operation data to the hard disk storage through the DPU and the Host, reducing the number of times the operator operation data is moved between the DPU, the Host and the hard disk, saving the time and system resources for moving the operator operation data, and improving the data processing efficiency of the DPU.

[0029] The specific implementation of the embodiments of the present application is further explained below in combination with the embodiments of the present application and the accompanying drawings.

[0030] Figure 1 FIG. 1 is a flow chart of a data processing method based on a DPU (ie, a local DPU) according to an embodiment of the present application. Specifically, Figure 1 As shown, the method can be executed by a local server, and the data processing method includes:

[0031] S11: Acquire operator operation data of a data reorganization operator and preset data storage conditions, where the preset data storage conditions at least include a total storage space and an overflow coefficient of each storage device that performs a mapping write operation to store the operator operation data.

[0032] Specifically, the operator operation data of the data reorganization operator is the data for executing the mapping operation and the reduction operation of the Shuffle operator, and the data reorganization operator has been unloaded to the DPU (i.e., the local DPU). The preset data storage conditions include the total storage space and overflow coefficient of the DPU, the total storage space and overflow coefficient of the Host, and the total storage space and overflow coefficient of the hard disk.

[0033] S12: Based on the total storage space and overflow coefficient of each storage device that performs the mapping write operation in the preset data storage condition, the available storage space of each storage device that performs the mapping write operation is calculated.

[0034] The operator operation data and the preset data storage conditions are obtained through the remote DPU that performs the mapping write operation, and the storage space of each storage device that performs the mapping write operation is calculated, for example, the total storage space of the remote DPU that performs the mapping write operation is 1G, the overflow coefficient is 80%, and the storage space of the DPU is calculated to be 0.8G; the total storage space of the remote Host that performs the mapping write operation is 1G, the overflow coefficient is 80%, and the storage space of the Host is also calculated to be 0.8G; the total storage space of the hard disk that performs the mapping write operation is 10G, the overflow coefficient is 100%, and the storage space of the Host is calculated to be 10G, wherein the overflow coefficient of each storage device that performs the mapping write operation can be adjusted according to actual usage, and the total storage space of each storage device that performs the mapping write operation is also adjusted with parameters such as the model of the storage device.

[0035] Since storage space for the remaining data needs to be reserved in each storage device that performs the mapping write operation, an overflow coefficient is set. When the stored operator calculation data exceeds the storage space available on a certain storage device, the remaining data of the mapping operation result is written to the remaining storage devices for storage to ensure the normal use of the storage device.

[0036] S13: Write the operator operation data to each storage device that performs the mapping write operation, so that the storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted. For example, each storage device includes a remote DPU, a remote host, and at least two storage devices of a local hard disk and a remote hard disk. For another example, the remote DPU and remote host that perform the mapping write operation, and the remote hard disk can be configured in a remote server; the local DPU and local host that perform the reduction read operation, and the local hard disk are configured in a local server.

[0037] It can be understood that in the mapping write operation stage, the operator calculation data is first moved from the remote DPU that performs the mapping write operation to the remote Host that performs the mapping write operation according to the partition, and then moved to the hard disk that performs the mapping write operation. Finally, the operator calculation data is written to the hard disk for storage. Regardless of whether the host Host has sufficient memory space, the operator calculation data must be written to the hard disk for storage.

[0038] Furthermore, in the reduction read operation phase, the hard disk that performs the reduction read operation first reads the operator operation data stored in the hard disk that performs the mapping write operation, then moves the data to the Host that performs the reduction read operation, and finally moves it again to the DPU that performs the reduction read operation for subsequent calculations. Therefore, during the operator execution process, the remote DPU that performs the mapping write operation and the DPU that performs the reduction read operation transmit the operator operation data. Multiple data movement operations are involved, resulting in a large time consumption for the operator operation data movement and low data processing efficiency of the DPU.

[0039] For example, Figure 2 The following is a data flow diagram of an operator operation for executing a Shuffle operator according to an embodiment of the present application.

[0040] like Figure 2 As shown, when the operator operation data needs to be stored in the mapping write operation phase, the operator operation data is preferentially written to the remote DPU that performs the mapping write operation for storage, so that the DPU that performs the reduce read operation in the reduce read operation phase can directly read the operator operation data stored in the remote DPU that performs the mapping write operation. Compared with the hard disk that performs the reduce read operation reading the operator operation data stored in the hard disk that performs the mapping write operation, and moving the operator operation data to the DPU that performs the reduce read operation via the hard disk that performs the reduce read operation and the Host that performs the reduce read operation, two data movement operations can be saved.

[0041] Next, when the storage space of the remote DPU that performs the mapping write operation is insufficient (that is, overflow will occur when continuing to write the operator operation data), the operator operation data will be written to the remote Host that performs the mapping write operation for storage, so that the DPU that performs the reduce read operation in the reduce read operation phase can first read the operator operation data stored by the remote Host that performs the mapping write operation, and move the operator operation data through the Host that performs the reduce read operation. Compared with the hard disk that performs the reduce read operation reading the operator operation data stored by the hard disk that performs the mapping write operation, and moving the operator operation data to the DPU that performs the reduce read operation through the hard disk that performs the reduce read operation and the Host that performs the reduce read operation, one data movement operation can be saved.

[0042] Finally, when the storage space of the Host that performs the mapping write operation does not meet the requirements (that is, overflow will occur when continuing to write operator calculation data), the remaining data of the mapping operation result will be written to the hard disk that performs the mapping write operation.

[0043] In this way, the number of times operator calculation data is moved between the DPU, Host and hard disk can be reduced, saving time and system resources for operator calculation data movement, thereby improving the data processing efficiency of the DPU.

[0044] In some embodiments, the operator operation data is written to each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, including: determining whether the amount of data of the mapping operation result is less than or equal to the storage space of the remote DPU that performs the mapping write operation; when the amount of data is less than or equal to the storage space of the remote DPU that performs the mapping write operation, writing the operator operation data to the remote DPU that performs the mapping write operation.

[0045] For example, when the operator operation data contains x pieces of data, namely D1..Da, and the total data size is 0.7G, the total storage space of the remote DPU that performs the mapping write operation is 1G, and the overflow coefficient is 80%, the calculated storage space of the DPU is 0.8G. At this time, the data volume of the mapping operation result is less than the storage space of the remote DPU that performs the mapping write operation. In the mapping write operation stage, the x pieces of operator operation data can be written into the remote DPU that performs the mapping write operation for storage. Figure 2 ,In the reduction read operation phase, the DPU that performs the reduction read operation can directly read the x operator operation data stored by the remote DPU that performs the mapping write operation for subsequent processing.

[0046] Furthermore, in some embodiments, the operator operation data is written to each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, and further includes:

[0047] When the amount of data is greater than the available storage space of the remote DPU that performs the mapping write operation, and is less than or equal to the sum of the available storage space of the remote DPU that performs the mapping write operation and the available storage space of the remote Host that performs the mapping write operation, the operator calculation data is preferentially written to the remote DPU that performs the mapping write operation until the available storage space of the remote DPU that performs the mapping write operation is full; the remaining data in the mapping operation result is written to the remote Host that performs the mapping write operation until the available storage space of the remote Host that performs the mapping write operation is full.

[0048] For example, when the operator operation data contains y data, namely D1..Da...Db, the total data size is 1.6G, of which the data size before Da is 0.8G, and the data size between Da and Db is 0.8G. The total storage space of the remote DPU that performs the mapping write operation is 1G, the overflow coefficient is 80%, and the calculated storage space of the DPU is 0.8G. The total storage space of the remote Host that performs the mapping write operation is also 1G, the overflow coefficient is also 80%, and the calculated storage space of the Host is also 0.8G.

[0049] Because the total amount of data calculated by the operator is greater than the storage space of the remote DPU that performs the mapping write operation, which is 0.8G, and is equal to the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, which is 1.6G.

[0050] Therefore, in the mapping write operation stage, the operator operation data before Da among the y operator operation data can be preferentially written into the remote DPU that performs the mapping write operation for storage, and the operator operation data between Da and Db can be written into the remote Host that performs the mapping write operation for storage.

[0051] Continue to refer to Figure 2 In the reduce read operation stage, the DPU performing the reduce read operation can directly read the operator operation data before Da stored by the remote DPU performing the mapping write operation, and after the Host performing the reduce read operation reads the operator operation data between Da and Db stored by the remote Host performing the mapping write operation, the operator operation data between Da and Db is moved through the Host performing the reduce read operation for subsequent processing.

[0052] It should be noted that if the total storage space of the remote DPU or Host that performs the mapping write operation is used to store the operator operation data, it may affect the storage of the remaining data, resulting in the use function of the DPU or Host being affected. Therefore, this application adapts the data volume of the operator operation data to the storage space available for each storage device that performs the mapping write operation to avoid affecting the use function of each storage device that performs the mapping write operation.

[0053] Furthermore, in some embodiments, the operator operation data is written to each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, and further includes:

[0054] When the amount of data is greater than the sum of the available storage space of the remote DPU that performs the mapping write operation and the available storage space of the remote Host that performs the mapping write operation, and is less than or equal to the sum of the available storage space of each storage device that performs the mapping write operation, the operator calculation data is preferentially written to the remote DPU that performs the mapping write operation until the available storage space of the remote DPU that performs the mapping write operation is full; the remaining data in the mapping operation result is preferentially written to the Host that performs the reduce read operation, and after the available storage space of the Host that performs the reduce read operation (local Host) is full, the remaining data except the data stored in the Host is written to the hard disk that performs the mapping write operation.

[0055] For example, when the operator operation data contains z data, which are D1..Da...Db...Dn, the total data volume is 10G, the data volume before Da is 0.8G, the data volume between Da and Db is 0.8G, the data volume between Db and Dn is 8.4G, the storage space of the remote DPU that performs the mapping write operation is 0.8G, the storage space of the remote Host that performs the mapping write operation is also 0.8G, and the storage space of the hard disk that performs the mapping write operation is 10G.

[0056] Because the total amount of data calculated by the operator is greater than the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, which is 1.6G, and is less than the sum of the storage space of each storage device that performs the mapping write operation, which is 11.6G.

[0057] Therefore, in the mapping write operation stage, the operator operation data before Da among the z operator operation data can be preferentially written to the remote DPU that performs the mapping write operation for storage, and the operator operation data between Da and Db can be written to the remote Host that performs the mapping write operation for storage, and the operator operation data between Db and Dn can be written to the hard disk that performs the mapping write operation for storage.

[0058] Continue to refer to Figure 2 In the reduction read operation stage, the DPU performing the reduction read operation can directly read the operator operation data before Da stored by the remote DPU performing the mapping write operation, and after the Host performing the reduction read operation reads the operator operation data between Da and Db stored by the remote Host performing the mapping write operation, the operator operation data between Da and Db is moved through the Host performing the reduction read operation. Similarly, after the hard disk performing the reduction read operation reads the operator operation data between Db and Dn stored by the hard disk performing the mapping write operation, the operator operation data between Db and Dn is moved through the hard disk performing the reduction read operation and the Host performing the reduction read operation for subsequent processing.

[0059] For the Shuffle operator, the present application does not need to write all operator operation data to the hard disk storage. A three-layer storage overflow solution of DPU, Host and hard disk is adopted. DPU storage is considered first. If the storage space of DPU is full of operator operation data, but the operator operation data is still remaining, it overflows to Host for storage. If the storage space of Host is full of operator operation data, but the operator operation data is still remaining, it overflows to the hard disk for storage. This greatly reduces the number of data movement times during the execution process of the Shuffle operator, and the data will overflow to the hard disk only when the data volume is very large.

[0060] In this way, based on the size relationship between the operator calculation data and the storage space available for each storage device that performs the mapping write operation, the operator calculation data is stored in the corresponding storage device to reduce the number of times the operator calculation data is moved between the DPU, Host, and hard disk, thereby reducing network transmission time and improving the efficiency of the Shuffle process.

[0061] In some embodiments, the DPU-based data processing method further includes: the DPU that performs the reduction read operation moves the operator operation data stored in the remote DPU that performs the mapping write operation through a parallel computing communication protocol.

[0062] The parallel computing communication protocol can be the nvlink protocol (a high-speed interconnect technology), which can be used to connect the DPU and other devices, such as the CPU, other DPUs or storage devices. nvlink has a high speed and is mainly used to accelerate tasks such as high-performance computing, machine learning and deep learning. It can achieve direct memory access between DPUs. The protocol achieves higher bandwidth and lower latency, which can improve data transmission efficiency.

[0063] Specifically, in the related art, during the mapping write operation stage, the operator calculation data is moved from the remote DPU that performs the mapping write operation to the remote Host that performs the mapping write operation, and then moved to the hard disk that performs the mapping write operation, and finally the operator calculation data is written to the hard disk for storage. During the reduction read operation stage, the operator calculation data stored on the hard disk that performs the mapping write operation is first read to the hard disk that performs the reduction read operation, and then moved to the Host that performs the reduction read operation, and finally moved again to the DPU that performs the reduction read operation for subsequent calculations. Among them, the hard disk that performs the reduction read operation and the hard disk that performs the mapping write operation transmit data through the network communication protocol. Due to the large network transmission delay and the large number of transfers between the DPU, Host and hard disk, the processing speed of the Shuffle process is affected.

[0064] Based on this, continue to refer to Figure 2, based on the adoption of a three-layer storage overflow solution of DPU, Host and hard disk to reduce the number of migrations, this application also makes corresponding adjustments to the communication protocols between various storage devices. In the reduction read operation phase, since the operator operation data is stored in the DPU, and the DPUs are connected through nvlink, the DPU performing the reduction read operation directly reads the operator operation data before Da stored by the remote DPU performing the mapping write operation through the nvlink protocol.

[0065] Furthermore, in some embodiments, the DPU-based data processing method also includes: the Host device that performs the reduce read operation moves the operator operation data stored in the remote Host device that performs the mapping write operation through the memory access communication protocol; and moves the operator operation data in the Host device to the DPU that performs the reduce read operation.

[0066] The memory access communication protocol is the RDMA (Remote Direct Memory Access) protocol, which allows network devices to directly access the memory of another device without going through the network protocol stack of the operating system. RDMA can be implemented on network technologies such as InfiniBand and Ethernet (RoCE). Its speed can usually reach tens of gigabits per second (Gbps) to hundreds of gigabits per second (Gbps), depending on the configuration of network devices and links. RDMA is mainly used in scenarios such as high-performance computing, cloud computing, and big data, and can provide high-throughput and low-latency data transmission.

[0067] Continue to refer to Figure 2 Since the operator operation data is also stored in the Host, and the Hosts are connected through RDMA, in the reduction read operation phase, the DPU performing the reduction read operation directly reads the operator operation data before Da stored by the remote DPU performing the mapping write operation through the nvlink protocol, and first reads the operator operation data between Da and Db stored by the remote Host performing the mapping write operation to the Host performing the reduction read operation through the RDMA protocol, and then moves the operator operation data between Da and Db stored by the Host performing the reduction read operation for subsequent processing.

[0068] In this way, by adjusting the data transmission protocol and adopting a communication protocol with higher data transmission efficiency to accelerate data transmission, the time consumption of moving operator calculation data can be reduced, thereby improving the efficiency of moving operator calculation data.

[0069] Furthermore, in some embodiments, the DPU-based data processing method also includes: the hard disk performing the reduce read operation moves the operator operation data stored in the hard disk performing the mapping write operation through a network communication protocol; and the operator operation data in the hard disk is moved to the DPU performing the reduce read operation via the Host device performing the reduce read operation.

[0070] As a common local area network technology, Ethernet is usually used to connect computers, network devices and other devices. The standard Ethernet speed usually ranges from hundreds of megabits per second (Mbps) to dozens of gigabits per second (Gbps), and is mostly used for ordinary data transmission and network connections.

[0071] Reference Figure 2 Since the operator operation data is also stored in the hard disk, and the data is transmitted between the hard disks via Ethernet, therefore, in the reduction read operation stage, the DPU performing the reduction read operation directly reads the operator operation data before Da stored by the remote DPU performing the mapping write operation through the nvlink protocol; and first reads the operator operation data between Da and Db stored by the remote Host performing the mapping write operation to the Host performing the reduction read operation through the RDMA protocol, and then moves the operator operation data between Da and Db stored by the Host performing the reduction read operation; and, reads the operator operation data between Db and Dn stored by the hard disk performing the mapping write operation through Ethernet to the hard disk performing the reduction read operation, and moves the operator operation data between Db and Dn via the hard disk performing the reduction read operation and the Host performing the reduction read operation for subsequent processing.

[0072] In summary, for the Shuffle operator, the present application does not need to write all operator operation data to the hard disk storage. In the mapping write operation stage, the DPU, Host and hard disk three-layer storage overflow solution is adopted. Based on the size relationship between the operator operation data and the storage space of each storage device that performs the mapping write operation, the operator operation data is stored in the corresponding storage device to reduce the number of times the operator operation data is moved between the DPU, Host and hard disk, thereby reducing the network transmission time and improving the efficiency of the Shuffle process. At the same time, in the reduction read operation stage, a corresponding network transmission solution is provided for each layer of the three-layer storage, and a communication protocol with higher data transmission efficiency is used to accelerate the data transmission, reduce the time consumption of moving the operator operation data, and thus improve the efficiency of moving the operator operation data. Among them, in the mapping write operation stage, each DPU and Host can communicate through the system bus protocol, and the Host and the hard disk can communicate through the peripheral bus protocol. The same is true for the reduction read operation stage.

[0073] The DPU-based data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc.

[0074] Figure 3 The present invention is a structural block diagram of a DPU-based data processing device according to an embodiment of the present application.

[0075] like Figure 3 As shown, the DPU-based data processing device 300 can execute Figure 1 The DPU-based data processing method, the DPU-based data processing device 300 includes: an acquisition module 301, which acquires the operator operation data of the data reorganization operator and the preset data storage conditions, the preset data storage conditions at least include the total storage space and overflow coefficient of each storage device that performs the mapping write operation to store the operator operation data; a calculation module 302, based on the total storage space and overflow coefficient of each storage device that performs the mapping write operation in the preset data storage conditions, calculates the storage space of each storage device that performs the mapping write operation; a writing module 303, which writes the operator operation data into each storage device that performs the mapping write operation, so that the storage space of each storage device that performs the mapping write operation and the data amount in the operator operation data are adapted.

[0076] In some embodiments of the present application, the write module 303 is specifically used to: determine whether the data amount of the mapping operation result is less than or equal to the storage space of the remote DPU that performs the mapping write operation; when the data amount is less than or equal to the storage space of the remote DPU that performs the mapping write operation, write the operator operation data to the remote DPU that performs the mapping write operation.

[0077] In some embodiments of the present application, the write module 303 is specifically used to: when the amount of data is greater than the storage space available for storage of the remote DPU that performs the mapping write operation, and is less than or equal to the sum of the storage space available for storage of the remote DPU that performs the mapping write operation and the storage space available for storage of the remote Host that performs the mapping write operation, the operator calculation data is preferentially written to the remote DPU that performs the mapping write operation until the storage space available for storage of the remote DPU that performs the mapping write operation is full; and the remaining data in the mapping operation result is written to the remote Host that performs the mapping write operation until the storage space available for storage of the remote Host that performs the mapping write operation is full.

[0078] In some embodiments of the present application, the write module 303 is specifically used to: when the amount of data is greater than the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, and is less than or equal to the sum of the storage space of each storage device that performs the mapping write operation, the operator calculation data is preferentially written to the remote DPU that performs the mapping write operation until the storage space of the remote DPU that performs the mapping write operation is full; the remaining data in the mapping operation result is preferentially written to the Host that performs the reduce read operation, and after the storage space of the Host that performs the reduce read operation is full, the remaining data in the mapping operation result is written to the hard disk that performs the mapping write operation.

[0079] In some embodiments of the present application, a reading module (not shown in the figure) is used for: a DPU that performs a reduction read operation reads operator operation data stored in a remote DPU that performs a mapping write operation through a parallel computing communication protocol.

[0080] In some embodiments of the present application, the reading module (not shown in the figure) is also used for: the Host device that performs the reduction read operation reads the operator operation data stored in the remote Host device that performs the mapping write operation through the memory access communication protocol; the moving module (not shown in the figure) is also used for: moving the operator operation data in the Host device that performs the reduction read operation to the DPU that performs the reduction read operation.

[0081] In some embodiments of the present application, the reading module (not shown in the figure) is also used for: the hard disk performing the reduction read operation reads the operator operation data stored in the hard disk performing the mapping write operation through the Ethernet network communication protocol; the moving module (not shown in the figure) is also used for: moving the operator operation data stored in the hard disk via the Host device performing the reduction read operation to the DPU performing the reduction read operation.

[0082] According to the DPU-based data processing device of the embodiment of the present application, the available storage space of each storage device that performs the mapping write operation is calculated to write the operator operation data into each storage device that performs the mapping write operation, so that the available storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, thereby eliminating the need to move all the operator operation data to the hard disk storage through the DPU and the Host, reducing the number of times the operator operation data is moved between the DPU, the Host and the hard disk, saving the time and system resources for moving the operator operation data, and improving the data processing efficiency of the DPU.

[0083] The DPU-based data processing device 300 of this embodiment is used to implement the corresponding DPU-based data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the DPU-based data processing device 300 of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0084] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0085] like Figure 4 As shown, the electronic device 400 may include: a processor 402, a memory 403, a communication interface 404 and a communication bus 405. The processor 402, the memory 403 and the communication interface 404 communicate with each other through the communication bus 405; the memory 403 is used to store at least one executable instruction, and the executable instruction enables the processor 402 to perform the operation corresponding to the DPU-based data processing method of any of the above embodiments.

[0086] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0087] The memory 403 is used to store programs, and may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0088] According to the electronic device of the embodiment of the present application, the available storage space of each storage device that performs the mapping write operation is calculated to write the operator operation data into each storage device that performs the mapping write operation, so that the available storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, thereby eliminating the need to move all the operator operation data to the hard disk storage through the DPU and the Host, reducing the number of times the operator operation data is moved between the DPU, the Host and the hard disk, saving the time and system resources for moving the operator operation data, and improving the data processing efficiency of the DPU.

[0089] An embodiment of the present application also provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the DPU-based data processing method of any of the above embodiments is implemented.

[0090] According to the computer storage medium of the embodiment of the present application, the available storage space of each storage device that performs the mapping write operation is calculated to write the operator operation data into each storage device that performs the mapping write operation, so that the available storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted, thereby eliminating the need to move all the operator operation data to the hard disk storage through the DPU and the Host, reducing the number of times the operator operation data is moved between the DPU, the Host and the hard disk, saving the time and system resources for moving the operator operation data, and improving the data processing efficiency of the DPU.

[0091] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0092] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the DPU-based data processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the DPU-based data processing method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the DPU-based data processing method shown herein.

[0093] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0094] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims. The systems, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

Claims

1. A data processing method based on DPU, characterized in that: The method comprises: Acquire operator operation data of the data reorganization operator and preset data storage conditions, wherein the preset data storage conditions at least include a total storage space and an overflow coefficient of each storage device that performs a mapping write operation and stores the operator operation data; Based on the total storage space and overflow coefficient of each storage device that performs the mapping write operation in the preset data storage condition, the available storage space of each storage device that performs the mapping write operation is calculated; The operator operation data is written into each storage device that performs the mapping write operation, so that the storage space of each storage device that performs the mapping write operation is adapted to the data amount in the operator operation data.

2. The DPU-based data processing method according to claim 1, characterized in that: The step of writing the operator operation data into each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation and the amount of data in the operator operation data are adapted includes: Determine whether the data volume of the mapping operation result of the data reorganization operator is less than or equal to the storage space of the remote DPU that performs the mapping write operation; When the data amount is less than or equal to the storage space of the remote DPU that performs the mapping write operation, the operator operation data is written into the remote DPU that performs the mapping write operation.

3. The DPU-based data processing method according to claim 2, characterized in that: The step of writing the operator operation data into each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation is adapted to the amount of data in the operator operation data further includes: When the data amount of the mapping operation result is greater than the storage space available for storage of the remote DPU that performs the mapping write operation, and is less than or equal to the sum of the storage space available for storage of the remote DPU that performs the mapping write operation and the storage space available for storage of the remote Host that performs the mapping write operation, the data amount of the mapping operation result is preferentially written into the remote DPU that performs the mapping write operation until the storage space available for storage of the remote DPU that performs the mapping write operation is full; The remaining data of the data amount of the mapping operation result is written into the remote host that performs the mapping write operation until the storage space of the remote host that performs the mapping write operation is full.

4. The DPU-based data processing method according to claim 3, characterized in that: The step of writing the operator operation data into each storage device that performs the mapping write operation so that the storage space of each storage device that performs the mapping write operation is adapted to the amount of data in the operator operation data further includes: When the data volume of the mapping operation result is greater than the sum of the storage space of the remote DPU that performs the mapping write operation and the storage space of the remote Host that performs the mapping write operation, and is less than or equal to the sum of the storage spaces of the storage devices that perform the mapping write operation, the data volume of the mapping operation result is preferentially written to the remote DPU that performs the mapping write operation until the storage space of the remote DPU that performs the mapping write operation is full; The remaining data of the data volume of the mapping operation result is preferentially written to the Host that performs the reduce read operation, and after the storage space of the Host that performs the reduce read operation is full, the remaining data except the data stored in the Host is written to the hard disk that performs the mapping write operation.

5. The DPU-based data processing method according to claim 1, characterized in that: The method further comprises: The DPU that performs the reduction read operation reads the operator operation data stored by the remote DPU that performs the mapping write operation through the parallel computing communication protocol.

6. The DPU-based data processing method according to claim 5, characterized in that: The method further comprises: The host device that performs the reduction read operation reads the operator operation data stored in the remote host device that performs the mapping write operation through the memory access communication protocol; The operator calculation data in the Host device that performs the reduce read operation is moved to the DPU that performs the reduce read operation.

7. The DPU-based data processing method according to claim 6, characterized in that: The method further comprises: The hard disk performing the reduction read operation reads the operator operation data stored in the hard disk performing the mapping write operation through the network transmission protocol; The operator calculation data in the hard disk that performs the reduce read operation is moved to the DPU that performs the reduce read operation via the host device that performs the reduce read operation.

8. A data processing device based on DPU, characterized in that: include: An acquisition module, which acquires operator operation data of a data reorganization operator and preset data storage conditions, wherein the preset data storage conditions at least include a total storage space and an overflow coefficient of each storage device that performs a mapping write operation and stores the operator operation data; A calculation module, based on the total storage space and overflow coefficient of each storage device that performs the mapping write operation in the preset data storage condition, calculates the storage space available for each storage device that performs the mapping write operation; A writing module writes the operator operation data into each storage device that performs the mapping write operation, so that the storage space of each storage device that performs the mapping write operation is adapted to the data amount in the operator operation data.

9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the DPU-based data processing method as described in any one of claims 1-7.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the DPU-based data processing method as described in any one of claims 1 to 7 is implemented.

Images