Data processing method, electronic equipment, readable storage medium and computing system

Abstract

The invention relates to the technical field of information, in particular to a data processing method, electronic equipment, a readable storage medium and a computing system. The method is applied to a computing system comprising a plurality of hosts, when data are transmitted among different heterogeneous computing units in the computing system, a globally unique identifier is generated for a storage space of data to be transmitted, and other heterogeneous computing units in a distributed system can obtain corresponding data to be transmitted through the globally unique identifier. Therefore, different heterogeneous computing units do not need to synchronously transmit data, the resource utilization rate of the heterogeneous computing units is improved, data does not need to be transferred through a processor, and the data transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a data processing method, an electronic device, a readable storage medium, and a computing system. Background Art

[0002] At present, the structure of neural network models is evolving towards large-scale, multimodal, sparse, asymmetric, irregular, etc. The training of such models often adopts a variety of parallel strategies. For example, a training task will be divided into multiple subtasks, each of which is executed on different heterogeneous computing units, and when necessary, the data generated by the calculation is transferred between the heterogeneous computing units where the subtasks are located, so as to complete the entire computing task. Among them, heterogeneous computing units refer to other processing units other than the central processing unit (CPU) (for example, graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPU), etc.). Devices including CPU and other heterogeneous computing units are heterogeneous devices. In the entire computing task, data transfer between computing subtasks (that is, data flow between heterogeneous computing units) often affects the efficiency of the entire computing task and the utilization of resources of heterogeneous computing units.

[0003] At present, the data flow between heterogeneous computing units is generally to specify the various subtask processes (hereinafter referred to as processes) that need to participate in the communication in the program of the computing task to form a communication group. Then, each process exchanges data to complete the entire communication process. However, in the above-mentioned data flow between heterogeneous computing units, when the processes of each subtask form a communication group, the subtask process is already running on the corresponding heterogeneous computing unit. In this way, even if the subtask on a heterogeneous computing unit needs to wait for the computing data of another heterogeneous computing unit before starting to execute, the subtask on the heterogeneous computing unit will always occupy the resources of the heterogeneous computing unit (for example, when the GPU needs to exchange data to enter the communication group, the computing resources of the GPU have been occupied). In other words, although the subtask on the heterogeneous computing unit is not actually running, it will always occupy the resources of the heterogeneous computing unit, resulting in a low overall utilization rate of the resources of the heterogeneous computing unit. Summary of the invention

[0004] In view of this, the present application provides a data processing method, an electronic device, a readable storage medium, and a computing system.

[0005] In a first aspect, the present application provides a data processing method, which is applied to a computing system, wherein the computing system includes a first host and a second host, the first host includes a first heterogeneous computing unit, and the second host includes a second heterogeneous computing unit; the method includes: obtaining data to be processed; assigning processing tasks of the data to be processed to the first host and the second host, wherein the first intermediate data generated by the first heterogeneous computing unit of the first host corresponding to the data to be processed is the intermediate data required by the second heterogeneous computing unit of the second host to complete the processing tasks corresponding to the data to be processed; wherein assigning the processing tasks of the data to be processed to the first host and the second host includes: setting the first intermediate data generated by the first heterogeneous computing unit corresponding to the data to be processed to have a first mapping relationship and a second mapping relationship, wherein the first mapping relationship is a mapping relationship between a first storage space for storing the first intermediate data in the first host and an identifier of the first intermediate data, and the second mapping relationship is a mapping relationship between the identifier of the first intermediate data and the first host; the second heterogeneous computing unit of the second host obtains the first intermediate data from the first storage space in the first host based on the first mapping relationship and the second mapping relationship, and completes the processing tasks corresponding to the data to be processed based on the first intermediate data.

[0006] That is to say, the first intermediate data generated by the first heterogeneous computing unit corresponding to the data to be processed needs to be transmitted to the second heterogeneous computing unit. The second host can determine that the first storage space storing the first intermediate data is located in the first host based on the second mapping relationship. And the first host can determine the first storage space according to the first mapping relationship. Therefore, the second host can determine the first storage space and transmit the first intermediate data in the first storage space to the second heterogeneous computing unit, thereby completing the data transmission between the first heterogeneous computing unit and the second heterogeneous computing unit. The first heterogeneous computing unit and the second heterogeneous computing unit do not need to transmit data synchronously, thereby improving the utilization rate of resources of the first heterogeneous computing unit and the second heterogeneous computing unit.

[0007] Exemplarily, in some other embodiments, the first intermediate data may also be flowable data, the first storage space storing the first intermediate data may also be a heterogeneous memory space hereinafter, and the identifier of the first intermediate data may also be a universally unique identifier (UUID) of a heterogeneous object corresponding to the heterogeneous memory space. It should be understood that the aforementioned heterogeneous object is the first storage space storing the first intermediate data.

[0008] In a possible implementation of the first aspect, the first storage space is a storage space in the memory of the first heterogeneous computing unit.

[0009] In a possible implementation of the first aspect above, the second heterogeneous computing unit of the second host obtains the first intermediate data from the first storage space in the first host based on the first mapping relationship and the second mapping relationship, including: the second host obtains the second mapping relationship from the host of the computing system based on the identifier of the first intermediate data; the second host obtains the first storage identifier of the first storage space from the first host based on the second mapping relationship; the first storage identifier includes a pointer to the first storage space, and / or an address of the first storage space; the second host transfers the first intermediate data to the second heterogeneous computing unit based on the first storage identifier.

[0010] That is to say, the second host can determine the first host storing the first intermediate data according to the second mapping relationship, and the first host determines that the first intermediate data is in the first storage space of the first heterogeneous computing unit in the first host based on the first mapping relationship. The first host sends the first storage identifier of the first storage space to the second host, and the second host can transmit the first intermediate data to the second heterogeneous computing unit based on the identifier of the first storage space. Therefore, the data flow between the first heterogeneous computing unit and the second heterogeneous computing unit does not need to be synchronized, thereby improving the utilization rate of the computing resources of the heterogeneous computing units.

[0011] In a possible implementation of the first aspect above, the second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, including: corresponding to the chip interconnection protocol supported between the first heterogeneous computing unit and the second heterogeneous computing unit, the second host sends the first storage identifier to the second heterogeneous computing unit; the second heterogeneous computing unit obtains the first intermediate data from the first storage space based on the chip interconnection protocol and the first storage identifier.

[0012] That is to say, the first heterogeneous computing unit and the second heterogeneous computing unit can directly share memory through the chip interconnection protocol, that is, the second heterogeneous computing unit can directly access the data of the first heterogeneous computing unit. When the second heterogeneous computing unit obtains the first storage identifier of the first storage space, it can access the first intermediate data in the first heterogeneous computing unit through the identifier of the first storage space. Therefore, there is no need to transfer data between the first heterogeneous computing unit and the second heterogeneous computing unit through the central processor, thereby improving the efficiency of data transmission.

[0013] Exemplarily, the first storage identifier is a pointer, and the second heterogeneous computing unit accesses the first storage space based on the pointer.

[0014] In a possible implementation of the first aspect above, the second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, and also includes: corresponding to the situation that the chip interconnection protocol is not supported between the first heterogeneous computing unit and the second heterogeneous computing unit and the first heterogeneous computing unit and the second heterogeneous computing unit support remote direct memory access; the second host allocates a second storage space in the second heterogeneous computing unit; the second heterogeneous computing unit obtains the first intermediate data from the first heterogeneous computing unit through remote direct memory access based on the first storage identifier, and stores the first intermediate data in the second storage space.

[0015] That is to say, the chip interconnection protocol is not supported between the first heterogeneous computing unit and the second heterogeneous computing unit, so the second heterogeneous computing unit cannot directly access the first heterogeneous computing unit through the first storage identifier. However, remote direct memory access is supported between the first heterogeneous computing unit and the second heterogeneous computing unit, so the second host can first allocate the second storage space in the second heterogeneous computing unit. The second heterogeneous computing can access the first storage space according to the first storage identifier, and copy the first intermediate data in the first storage space to the second storage space to complete the data flow. Therefore, the data flow between the first heterogeneous computing unit and the second heterogeneous computing unit does not require the central processor to transfer data, thereby improving the efficiency of data transmission.

[0016] In a possible implementation of the first aspect above, the first host also includes a first central processing unit and a first memory corresponding to the first central processing unit, and the second host also includes a second central processing unit and a second memory corresponding to the second central processing unit; and the second host transfers the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, and also includes: corresponding to the fact that the chip interconnection protocol is not supported between the first heterogeneous computing unit and the second heterogeneous computing unit and the first heterogeneous computing unit and the second heterogeneous computing unit do not support remote direct memory access; the second host allocates a second storage space in the second heterogeneous computing unit; the first central processing unit stores the first intermediate data in the first memory based on the first storage identifier; the first central processing unit transmits the first intermediate data in the first memory to the second memory through the network; and the second central processing unit transmits the first intermediate data from the second memory space to the second storage space.

[0017] That is to say, in some embodiments, the first heterogeneous computing unit and the second heterogeneous computing unit do not support chip interconnection protocol or remote direct memory access. Therefore, the first heterogeneous computing unit and the second heterogeneous computing unit can only transfer data through the first central processor in the first host and the second central processor in the second host, thereby ensuring that data can be transmitted between the first heterogeneous computing unit and the second heterogeneous computing unit.

[0018] It should be understood that the above-mentioned schemes for transmitting the first intermediate data between the first heterogeneous computing unit and the second heterogeneous computing unit (i.e., transmission based on chip interconnection protocol, transmission based on remote direct memory access, and transmission based on the first central processing unit and the second central processing unit) can be parallel, that is, the first heterogeneous computing unit and the second heterogeneous computing unit can support at least one of the above-mentioned data transmission methods. The first heterogeneous computing unit and the second heterogeneous computing unit can select a scheme to transmit the first intermediate data according to specific needs.

[0019] In a possible implementation of the first aspect, the chip interconnection protocol includes any one of the following protocols: a high-speed peripheral component interconnect interface, a computing fast link interface, or an NVLink communication protocol.

[0020] It should be understood that in other embodiments, the chip interconnection protocol may also be other protocols that support memory sharing between the first heterogeneous computing unit and the second heterogeneous computing unit, which will not be elaborated herein.

[0021] In a second aspect, the present application provides a computing system, the computing system comprising a first host and a second host, the first host comprising a first heterogeneous computing unit, the second host comprising a second heterogeneous computing unit; the first host and the second host are used to perform a processing task of the data to be processed acquired by the computing system, wherein the first intermediate data generated by the first heterogeneous computing unit of the first host corresponding to the data to be processed is the intermediate data required for the second heterogeneous computing unit of the second host to complete the processing task of the data to be processed; wherein the first host and the second host are used to perform the processing task of the data to be processed acquired by the computing system, including: the first host is used to set the first intermediate data generated by the first heterogeneous computing unit corresponding to the data to be processed to have a first mapping relationship and a second mapping relationship, wherein the first mapping relationship is a mapping relationship between a first storage space for storing the first intermediate data of the first host and an identifier of the first intermediate data, and the second mapping relationship is a mapping relationship between the identifier of the first intermediate data and the first host; the second host is used to obtain the first intermediate data from the first storage space in the first host through the second heterogeneous computing unit based on the first mapping relationship and the second mapping relationship, and complete the processing task of the data to be processed based on the first intermediate data. The beneficial effects that can be achieved in the second aspect can refer to the beneficial effects of the method provided in any embodiment of the first aspect, which will not be repeated here.

[0022] In a third aspect, the present application provides an electronic device, the electronic device comprising: a memory for storing instructions; and at least one processor for executing instructions so that the device implements the first aspect and any possible implementation method of the first aspect. The beneficial effects that can be achieved in the third aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed by a device, the computer implements the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the fourth aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0024] In a fifth aspect, the present application provides a computer program product, which, when running on a device, enables the device to implement the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the fifth aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1a A schematic diagram of a host in a distributed system is shown;

[0026] Figure 1b A schematic diagram showing data flow between heterogeneous computing units is shown;

[0027] Figure 2 An MPI collective communication process is shown;

[0028] Figure 3 shows a communication process of MPI;

[0029] Figure 4a An architectural diagram of a Rendezvous mechanism is shown;

[0030] Figure 4b A schematic diagram of transmitting data through a CPU is shown;

[0031] Figure 5 An interaction flow chart of the Rendezvous mechanism is shown;

[0032] Figure 6 A schematic diagram of data exchange between processes is shown;

[0033] Figure 7 According to an embodiment of the present application, an architecture diagram of a distributed system is shown;

[0034] Figure 8a According to some embodiments of the present application, an architecture diagram of a host is shown;

[0035] Figure 8b According to some embodiments of the present application, a schematic diagram of establishing a connection between an application and a heterogeneous computing unit is shown;

[0036] Figure 9aAccording to some embodiments of the present application, an interactive flow chart of creation and publication of heterogeneous objects is shown;

[0037] Figure 9b According to some embodiments of the present application, an interactive flow chart of a heterogeneous object reading and writing process is shown;

[0038] Fig.10 According to some embodiments of the present application, an architecture diagram of a distributed system is shown;

[0039] Fig.11 According to some embodiments of the present application, a schematic diagram of an application exchanging data is shown;

[0040] Fig.12 According to some embodiments of the present application, a flowchart of an implementation of an application program applying for heterogeneous memory is shown;

[0041] Fig.13 According to some embodiments of the present application, a flowchart for implementing heterogeneous object publishing is shown;

[0042] Fig.14 According to some embodiments of the present application, a flowchart for implementing an application program to obtain heterogeneous objects is shown;

[0043] Fig.15 According to some embodiments of the present application, a schematic structural diagram of a host is shown. DETAILED DESCRIPTION

[0044] Illustrative embodiments of the present application include, but are not limited to, a data processing method, an electronic device, a readable storage medium, and a computing system.

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Figure 1a A schematic diagram of a host in a distributed system is shown.

[0047] It should be understood that the distributed system includes multiple hosts, each of which includes at least one CPU and at least one heterogeneous computing unit, such as a GPU, an NPU, and a TPU, etc. In this embodiment, the host includes one CPU and three GPUs.

[0048] like Figure 1aAs shown, the host 100 includes a CPU 101, a GPU 102, a GPU 103, and a GPU 104. The application M running in the host 100 is divided into three processes. Process A runs on GPU 102, process B runs on GPU 103, and process C runs on GPU 104.

[0049] It should be understood that in this embodiment, the data flow between heterogeneous computing units can be, for example, data flow between GPU 102, GPU 103 and GPU 104. In other embodiments, the data flow between heterogeneous computing units can also be data flow between heterogeneous computing units on different hosts. It can be understood that the above-mentioned host 100 as a whole can be used as a heterogeneous device, or at least one heterogeneous computing unit in the above-mentioned host 100 and at least one CPU together constitute a heterogeneous device.

[0050] It should be understood that a distributed system includes multiple Figure 1a In the embodiment, each host 100 may include, for example, Figure 1a The host 100 of the embodiment may include more or fewer heterogeneous computing units, or may include different types of heterogeneous computing units.

[0051] It should be understood that in some embodiments of the present application, the host 100 may be, for example, a hardware device such as a mobile phone, a computer, or a server, or may be a virtual machine in an electronic device system.

[0052] Figure 1b A schematic diagram showing data flow between heterogeneous computing units is shown.

[0053] like Figure 1b As shown, the application M running in the host 100 is divided into three processes. Process A runs on GPU 102, process B runs on GPU 103, and process C runs on GPU 104. When necessary, the data generated by the calculation is transferred between each process to complete the operation of the entire application M. For example, the output data generated by the calculation of subprocess A needs to be transferred to process B as input, where the arrow represents the need for data transfer between processes. Similarly, the output data generated by process A also needs to be transferred to process C as input, and the output data generated by process B is then transferred to process A as input.

[0054] In some embodiments, a collective communication paradigm based on a message passing interface (MPI) model may be used to complete data flow between heterogeneous computing units.

[0055] For example, Figure 2 An MPI collective communication process is shown.

[0056] like Figure 2 As shown, when a program M is executed, three application processes need to transfer between heterogeneous computing units, for example, process A, process B and process C. Among them, process A executes five business logics A0, A1, A2, A3 and A4 in chronological order. Process B executes six business logics B0, B1, B2, B3, B4 and B5 in chronological order. Process C executes six business logics C0, C1, C2, C3, C4 and C5 in chronological order.

[0057] The application M may establish a communication group including process A, process B and process C, and process A, process B and process C have already occupied the resources of the corresponding heterogeneous computing units when the communication group is established.

[0058] Then, each process calls MPI_Init (the initialization function in the MPI model) to initialize the communication environment (MPIenvironment) and obtains its own serial number (rank) in the communication group. The serial number is used to identify each process. For example, the serial number of process A is 0, the serial number of process B is 1, and the serial number of process C is 2.

[0059] Finally, each process exchanges data, thereby completing the entire MPI communication process (MPI_Finalize). For example, the duration of the exchange of data is T. In the T1 time period, the B0 segment business logic of the B process with a sequence number of 1 exchanges data with the C1 segment business logic of the C process with a sequence number of 2. In the T2 time period, the A1 segment business logic of the A process with a sequence number of 0 exchanges data with the B2 segment business logic of the B process with a sequence number of 1. In the T3 time period, the A2 segment business logic of the A process with a sequence number of 0 exchanges data with the C4 segment business logic of the C process with a sequence number of 2. In the T4 time period, the B5 segment business logic of the B process with a sequence number of 1 exchanges data with the C5 segment business logic of the C process with a sequence number of 2. In the above embodiment, T can be, for example, 10s, and T1, T2, T3 and T4 are, for example, all 1s. In other embodiments, T, T1, T2, T3 and T4 can also be other time periods, and this application does not impose any restrictions on the above time periods.

[0060] From the above communication process based on the MPI model, it can be seen that the MIP model needs to establish a communication group for each process participating in the communication in advance, and then perform data communication.

[0061] It can be understood that the resource utilization rate of the above MPI model is low during the heterogeneous data flow. For example, when the above processes A, B, and C form a communication group, the resources of the corresponding heterogeneous devices are already occupied. While waiting for the data exchange to actually occur, the resources of these heterogeneous devices cannot be used by other computing tasks. For example, the time for the above entire data exchange is T = 10s, while the actual data exchange time is only T1 + T2 + T3 + T4 = 4s. Obviously, the time of T1 + T2 + T3 + T4 is less than the time of T. When the resources of the heterogeneous devices are occupied, there is 10s-4s = 6s without doing any work, resulting in a low overall utilization rate of the resources of the heterogeneous devices.

[0062] In addition, according to the provisions of MPI, the communication rules between processes are symmetrical. In other words, in the communication method based on MPI, only processes with the same communication rules can enter a communication group. If there are processes in the distributed system that use different communication rules, these processes cannot exchange data through MPI.

[0063] For example, Figure 3 The communication process of process 0 to process 7 is shown in FIG. Among them, process 1 to process 3 are applicable to the first communication rule. Process 4 and process 7 are applicable to the second communication rule. It can be understood that in the MPI communication process, process 1 to process 3 can form a first communication group and thus exchange data based on the first communication rule. Process 4 and process 7 can form a second communication group and thus exchange data based on the second communication rule. However, the processes of the first communication group and the processes of the second communication group cannot exchange data due to different communication rules, that is, MPI communication is not applicable.

[0064] It can be seen that in the above MPI model, in the communication process of heterogeneous computing units, each heterogeneous computing unit needs to form a communication group, resulting in low resource utilization of heterogeneous computing units. In addition, the MPI model is only suitable for establishing a communication group for processes with the same communication rules. If the communication rules between processes are different, it is no longer suitable to use the MPI model for communication. Therefore, the MPI model is not universal.

[0065] In other embodiments, in order to improve the utilization rate of resources of heterogeneous computing units, the Rendezvous mechanism (a data transmission method) can also be used to perform point-to-point (peer to peer, P2P) tensor transmission to exchange data between heterogeneous computing units.

[0066] It should be understood that the present embodiment uses tensors as data transmitted between heterogeneous computing units. In other embodiments, other types of data may be transmitted between heterogeneous computing units based on the Rendezvous mechanism.

[0067] Figure 4a A diagram showing an architecture of the Rendezvous mechanism.

[0068] The following introduces the Rendezvous mechanism by taking the data transmission between process 1 and process 2 as an example, where process 1 is the sender of the data transmission and process 2 is the receiver of the data.

[0069] It should be understood that process 1 and process 2 can run on the same CPU or on two CPUs respectively. When process 1 and process 2 run on two CPUs respectively, the two CPUs can be in the same host or in different hosts. This embodiment introduces that process 1 and process 2 run on two CPUs respectively, and the two CPUs are located in different hosts. Data is transmitted between the two hosts through remote procedure call (RPC).

[0070] like Figure 4a As shown, the Rendezvous mechanism mainly consists of two software components:

[0071] Sender 520: located in process 1, its main function is to place the tensors generated by process 1 (for example, tensors in GPU memory 510) into a tensor table 530 managed by process 1 for easy subsequent search.

[0072] It should be understood that tensor table 530 is a memory space in the memory of CPU A running process 1, which is used to store tensors.

[0073] Receiver 540: located in process 2, used to receive tensors and copy the received tensors to GPU memory 550.

[0074] It should be understood that when process 1 needs to transmit data with other processes, process 1 will establish another set of sender and receiver combinations with other processes, and will not communicate with other processes through sender 520. That is, the combination of sender 520 and receiver 540 can only be used for communication between process 1 and process 2.

[0075] It should be understood that when data needs to be transmitted between two processes, it is necessary to insert the combination of the sender 520 and the receiver 540 into the corresponding process to perform data transmission.

[0076] The following describes the data transmission process in the above-mentioned Rendezvous mechanism.

[0077] Figure 4b A schematic diagram showing data transmission via a CPU is shown.

[0078] like Figure 4b As shown, process 1 runs in CPU A, and tensor table 530 is stored in CPU A memory. CPU A, CPU A memory, and GPU memory 510 transmit data via bus 10. Process 2 runs in CPU B, and CPU B, CPU B memory, and GPU memory 550 transmit data via bus 20.

[0079] CPU A is configured with a network card 560, and CPU B is configured with a network card 570. CPU A communicates with the network card 570 of CPU B via the network card 560, thereby transmitting data.

[0080] Exemplarily, the sender 520 transfers the tensor in the GPU memory 510 to the tensor table 530 in the CPU A memory through the bus 10. When the receiver 540 needs to obtain the tensor, it sends the tensor identifier of the corresponding tensor to the sender 520. The sender 520 obtains the corresponding tensor from the tensor table 530 based on the tensor identifier, and then sends it to the network card 570 through the network card 560. CPU B stores the received tensor in the CPU B memory, and then the receiver 540 saves the tensor in the CPU B memory to the GPU 550, thereby completing the data flow.

[0081] Next, based on the above Rendezvous mechanism architecture, the Rendezvous communication process is introduced.

[0082] Figure 5 An interaction flow chart of the Rendezvous mechanism is shown.

[0083] like Figure 5 As shown, the process includes:

[0084] 501, the sender 520 obtains the tensor to be sent from the GPU memory 510.

[0085] Exemplarily, the tensor obtained by the sender 520 has a unique tensor identifier.

[0086] 502 , the sender 520 stores the tensor to be sent in the tensor table 530 .

[0087] Exemplarily, the tensor table 530 is a memory space in the memory of CPU A, which is used to store tensors. After the sender 520 stores the tensor in the tensor table 530 through the bus 10, the corresponding tensor can be found through the tensor identifier.

[0088] 503, the receiver 540 sends the tensor identifier of the tensor to be obtained to the sender 520 through a remote procedure call message.

[0089] Exemplarily, in a distributed system, process 1 can send a tensor identifier to process 2 by message passing. When process 2 needs to obtain the tensor corresponding to the tensor identifier, the tensor identifier can be sent to the sender 520 via an RPC message through the receiver 540 in process 2.

[0090] 504 , the sender 520 obtains the corresponding tensor from the tensor table 530 based on the tensor identifier of the tensor.

[0091] 505, the sender 520 sends the tensor to the receiver 540 in the form of an RPC message.

[0092] Exemplarily, when the receiver 540 obtains the tensor, it stores the tensor in the CPU B memory.

[0093] 506 , the receiver 540 stores the tensor data into GPU memory 550 .

[0094] It can be understood that the entire communication process of the Rendezvous mechanism in the above process is triggered by the receiver 540, and the sender 520 only needs to put the tensor into the tensor table 530 of the CPU A memory to wait for the request of the receiver 540. In other words, the processes of the sender 520 and the receiver 540 do not need to form a communication group, which can improve the resource efficiency of heterogeneous computing units compared to the method based on the MPI communication paradigm.

[0095] However, in the communication method based on the Rendezvous mechanism, it is necessary to insert a combination of sender 520 and receiver 540 between the subtasks participating in the communication (for example, process 1 and process 2), indicating that data exchange needs to occur between the subtasks, and sender 520 and receiver 540 are considered to be part of the computing task. If there are multiple processes, it is necessary to insert multiple combinations of senders and receivers.

[0096] For example, Figure 6 Figure 1 shows a schematic diagram of data exchange between processes. Figure 6 As shown, process A and process C exchange data through sender A1 and receiver B1, process A and process B exchange data through sender A2 and receiver B2, and process B and process D exchange data through sender A3 and receiver B3.

[0097] It can be understood that when the computing task is very complex, adopting this approach will require inserting a large number of sender and receiver combinations, increasing the complexity of task expression.

[0098] In addition, when tensors are transferred across processes based on the Rendezvous mechanism, the tensors will be copied multiple times, resulting in low data transmission efficiency.

[0099] For example, refer to Figure 4b , the sender 520 needs to copy the tensor from the GPU memory 510 to the CPU A memory, and then transmit the tensor to the CPU B memory of the receiving end (process 2) through the network through RPC, and then the receiving end 540 further copies the tensor from the CPU B memory to the GPU memory 550. This method that must be transferred through the CPU memory does not fully utilize the connection topology between GPU hardware and has poor performance.

[0100] It should be understood that in the above scheme, when data is transferred between heterogeneous computing units, there are problems such as low utilization of heterogeneous device resources, poor versatility of data transfer methods, and low efficiency.

[0101] In order to solve the problem of low resource utilization of heterogeneous computing units in the above-mentioned data flow method between heterogeneous computing units, the present application provides a data processing method, which is applied to a distributed system including multiple hosts. When the corresponding heterogeneous computing unit of each host generates data (hereinafter referred to as flowable data) that needs to be transferred to other heterogeneous computing units, a universally unique identifier (universally unique identifier, UUID) can be assigned to the heterogeneous memory space (hereinafter referred to as heterogeneous objects) corresponding to the flowable data, and a first mapping relationship between the UUID and the heterogeneous memory space is stored. In addition, the distributed system can store metadata (hereinafter referred to as flowable data information) of all heterogeneous objects in the distributed system, wherein the metadata includes the corresponding UUID of the heterogeneous object and the second mapping relationship between the host where the heterogeneous object is located.

[0102] When a heterogeneous computing unit needs to access the transferable data in a heterogeneous object, the metadata of the heterogeneous object can be obtained from the stored transferable data information according to the UUID of the heterogeneous object, and the host where the heterogeneous object is located is determined based on the second mapping relationship in the obtained metadata. Then, the transferable data in the heterogeneous object is accessed based on the first mapping relationship in the host.

[0103] Exemplarily, the storage address of the metadata can be obtained by, for example, hashing the UUID consistently. In a distributed system, the metadata of the heterogeneous objects corresponding to the UUID can be stored in the result of hashing the UUID consistently. It should be understood that the above-mentioned host can include, for example, a server, an electronic device such as a mobile phone or a computer, or a virtual machine in an operating system of an electronic device.

[0104] Through this solution, when data is transferred between heterogeneous computing units, there is no need to pre-synchronize and establish a communication group. Therefore, there is no need to wait for heterogeneous computing units to exchange data, thereby improving the resource utilization of heterogeneous computing units. Similarly, this solution is applicable to data transfer between various heterogeneous computing units and has strong applicability. Heterogeneous objects do not need to be copied multiple times, which improves the efficiency of data transfer between heterogeneous computing units.

[0105] The following introduces the distributed system in the embodiment of the present application.

[0106] Figure 7 According to an embodiment of the present application, an architecture diagram of a distributed system is shown.

[0107] It should be understood that a distributed system includes multiple hosts 100. In some embodiments, the host 100 may be a physical machine, such as a mobile phone, a computer, or a server. In other embodiments, the host 100 may be a virtual machine, such as a virtual machine in an operating system of a mobile phone, a computer, or a server.

[0108] For example, in some embodiments of the present application, the above-mentioned distributed system is introduced by taking the architecture of the host 100 and the host 100 ′ as an example.

[0109] like Figure 7 As shown, the distributed system includes a host 100 and a host 100'. It should be understood that similar modules are deployed on the host 100 and the host 100'.

[0110] The host 100 includes an application A1, a client interface B1 corresponding to the application A1, an application A2, a client interface B2 corresponding to the application A2, and a heterogeneous object management module 710 (hereinafter referred to as a worker). The client interface B1 and the client interface B2 may be referred to as a client module. The heterogeneous object management module 710 includes a heterogeneous object data module 711, a heterogeneous object metadata module 712, and a heterogeneous computing unit memory 713.

[0111] The heterogeneous object management module 710 is mainly responsible for mapping the heterogeneous memory space storing the flowable data in the heterogeneous computing unit memory 713 into a heterogeneous object, and assigning a UUID to the heterogeneous object. Workers on different hosts can transfer control messages and data to each other through RPC calls (RPC supports ordinary transmission control protocol (TCP), Internet protocol (IP) and remote direct memory access (RDMA) protocol transmission), thereby realizing distributed reading and writing of heterogeneous objects, that is, in a distributed system, workers on different hosts can obtain the flowable data corresponding to heterogeneous objects.

[0112] The heterogeneous object data module 711 is used to store and manage heterogeneous objects, and is responsible for the distributed reading and writing and concurrency control of heterogeneous objects. For example, the heterogeneous memory space corresponding to the flowable data in the heterogeneous computing unit memory 713 is mapped into a heterogeneous object, and a UUID is assigned to the heterogeneous object, a first mapping relationship between the UUID and the heterogeneous memory space is stored, and the UUID of the heterogeneous object is published to the distributed system for other applications in the distributed system to call.

[0113] The heterogeneous object metadata module 712 is used to store and manage the metadata of heterogeneous objects. The metadata of heterogeneous objects includes the mapping relationship between the UUID of the heterogeneous object and the host where the heterogeneous object is located. It should be understood that there is a worker running in each host of the distributed system, that is, there is a one-to-one correspondence between the worker and the host. Therefore, the host can be indicated by the worker identifier of the worker running in the host where the heterogeneous object is located, that is, the aforementioned second mapping relationship is represented by the mapping relationship between the UUID of the heterogeneous object and the worker identifier of the corresponding worker. For ease of description, the worker identifier of the worker running in the host where the heterogeneous object is located is referred to as the address of the heterogeneous object.

[0114] The heterogeneous object metadata module 712 can also perform addressing according to the metadata of the heterogeneous object, that is, determine the worker identifier corresponding to the heterogeneous object according to the UUID of the heterogeneous object, thereby realizing distributed addressing of the heterogeneous object.

[0115] Workers are used to read and write interfaces for heterogeneous objects in applications, thereby providing heterogeneous object management services to the outside world. For example, the UUID of a heterogeneous object is published to a distributed system, so that other hosts in the distributed system (or workers corresponding to other hosts) can access the heterogeneous object based on the UUID and obtain the transferable data in the heterogeneous object.

[0116] The client module encapsulates the basic interface provided by the worker and is deployed in the application process in the form of a library. The application interacts with the worker through the client module to achieve distributed reading and writing of heterogeneous memory data by the application.

[0117] In some embodiments, an application can be used to interact with a worker in the corresponding host, but not with workers in other hosts. For example, in host 100, application A1 only interacts with heterogeneous object management module 710 through client interface B1, and application A2 only interacts with heterogeneous object management module 710 through client interface B2. Application A1 in host 100 cannot interact with heterogeneous object management module 710' in host 100'.

[0118] The heterogeneous computing unit memory 713 is the memory corresponding to the heterogeneous computing unit in the host 100, such as the memory corresponding to the GPU. The host 100 includes a corresponding management toolkit to facilitate the upper-level software to implement basic management operations such as memory allocation and memory copy. The host 100 also needs to provide a cross-process shared access method. For example, the GPU and its corresponding management toolkit. For example, the compute unified device architecture (CUDA) toolkit, and the inter-process communication (IPC) mechanism of CUDA can realize cross-process sharing of GPU memory.

[0119] It should be understood that the host 100 in the embodiment of the present application may include, for example, a server, a mobile phone, a computer, and other physical machines. Taking the server as an example, the heterogeneous computing unit memory 713 is the memory corresponding to other processing units in the server that do not include the CPU, such as GPU, NPU, TPU, etc. Application A1 and application A2 are processes running on the server.

[0120] In other embodiments, the host 100 may also be a virtual machine in a system of an electronic device such as a server, a mobile phone, or a computer. Taking the virtual machine on the server as an example, the heterogeneous computing unit memory 713 is the memory of other processing units excluding the CPU allocated to the virtual machine by the device where the virtual machine is located, such as the memory corresponding to the GPU, NPU, or TPU. The heterogeneous computing unit memory 713 includes all the memories corresponding to all heterogeneous computing units running in the virtual machine, or part of the memory of the heterogeneous computing unit (that is, part of the memory of the heterogeneous computing unit on the server is managed by the heterogeneous object management unit 710 of the virtual machine). It should be understood that the heterogeneous computing units running in the virtual machine may be part of the heterogeneous computing units in the server, or all of the heterogeneous computing units in the server. The same heterogeneous computing unit may also run in different virtual machines of the server.

[0121] The host 100' includes modules similar to the host 100, namely, application A1', client interface B1' corresponding to application A1', application A2', client interface B2' corresponding to application A2', and heterogeneous object management module 710'. The heterogeneous object management module 710' includes a heterogeneous object data module 711', a heterogeneous object metadata module 712', and a heterogeneous device memory 713'. The heterogeneous object management module 710 of the host 100 and the heterogeneous object management module 710' of the host 100' can transfer control messages and data through RPC calls.

[0122] It should be understood that in other embodiments, the host 100' may also include modules different from the host 100, but the corresponding client module and the heterogeneous object management module 710' need to be deployed in the host 100'. In addition, each host in the distributed system at least includes a corresponding heterogeneous object management module and a client module, so as to complete the reading and writing of heterogeneous objects and provide heterogeneous object management services to the outside.

[0123] In the above-mentioned distributed system, each host includes a heterogeneous object management module and a client module. The application on the host can interact with the heterogeneous object management module based on the client module. The heterogeneous object management module can map the heterogeneous memory corresponding to the heterogeneous computing unit into a heterogeneous object and publish the UUID of the heterogeneous object, so that all heterogeneous object management modules in the distributed system can obtain the above-mentioned UUID. The heterogeneous object management modules between each host transmit control messages and data through RPC calls. Therefore, the heterogeneous object management modules on each host in the distributed system can obtain the address of the corresponding heterogeneous object based on the UUID of the heterogeneous object, and access the heterogeneous object according to the address of the heterogeneous object, and obtain the flowable data in the heterogeneous object, thereby realizing the data flow of the heterogeneous computing units in each host. The data flow between the above-mentioned heterogeneous computing units does not need to synchronously establish a communication group for multiple applications, thereby improving the resource utilization of the heterogeneous computing units, and the heterogeneous objects do not need to be transferred by copying to the CPU of the host, thereby improving the efficiency of data flow, and the above-mentioned data flow is suitable for data exchange between various heterogeneous computing units, and has strong applicability.

[0124] Based on the above distributed system, a host architecture is introduced below.

[0125] Figure 8a According to some embodiments of the present application, an architecture diagram of a host is shown.

[0126] It should be understood that the host in this embodiment takes a physical machine as an example, that is, it includes electronic devices such as servers, mobile phones and computers.

[0127] like Figure 8a As shown, the host 100 includes a software layer and a hardware layer. The software layer includes an application 810, a client module 820 deployed in the process of the application 810, and a heterogeneous object management module 710.

[0128] The heterogeneous object management module 710 includes a heterogeneous object data module 711 and a heterogeneous object metadata module 712. The heterogeneous object data module 711 includes a heterogeneous memory mapping module 831 and a communication management module 832. The heterogeneous object metadata module 712 includes a metadata storage module 833 and an addressing module 834.

[0129] The hardware layer includes CPU memory 850 , heterogeneous computing unit memory 713 , and network card 860 .

[0130] The heterogeneous memory mapping module 831 is used to manage the heterogeneous computing unit memory 713. For example, the heterogeneous memory space corresponding to the flowable data in the heterogeneous computing unit memory 713 is mapped into a heterogeneous object, and a UUID is assigned to the heterogeneous object. The heterogeneous memory mapping module 831 stores the mapping relationship between the memory space corresponding to the heterogeneous object and the corresponding UUID (i.e., the first mapping relationship between the UUID and the heterogeneous memory space).

[0131] The communication management module 832 is used to interact with the application 810 through the client module 820. For example, the communication management module 832 can receive a message from the client module 820 that the application 810 needs to allocate heterogeneous memory. And after the heterogeneous object management module 710 allocates the heterogeneous memory, the communication management module 832 can send the relevant information of the allocated heterogeneous memory to the client module 820. After the application 810 stores the flowable data in the heterogeneous memory, and the heterogeneous memory mapping module 831 maps the corresponding heterogeneous memory into a heterogeneous object and allocates a UUID, the communication management module 832 can return the UUID of the above heterogeneous object to the application 810 through the client module 820. Based on the message passing in the distributed system, the application 810 can send the UUID to other applications in the distributed system.

[0132] The metadata storage module 833 is used to store metadata of heterogeneous objects, for example, to store the mapping relationship between the UUID of the heterogeneous object and the address of the heterogeneous object (ie, the worker address).

[0133] The addressing module 834 is used to obtain the worker identifier of the metadata of the corresponding heterogeneous object based on the UUID of the heterogeneous object, and send a request to obtain the address of the heterogeneous object to the worker corresponding to the identifier.

[0134] Data can be transferred between the CPU memory 850 and the heterogeneous computing unit memory 713 .

[0135] The network card 860 can be used for communication between the host 100 and workers of other hosts. For example, corresponding network cards 860 are configured on different hosts, and workers on different hosts can perform RPC calls through their respective network cards 860 to transfer control messages and data.

[0136] Next, the application program in the host 100 is introduced.

[0137] It should be understood that the application 810 involved in the embodiment of the present application can be, for example, a set of logically related functions, responsible for executing corresponding computing tasks (some of which may need to be executed on heterogeneous computing units, thus requiring heterogeneous memory). Each application 810 runs on a host 100 as an independent process, and multiple applications 810 can be deployed on each host 100.

[0138] If each application 810 needs to use a heterogeneous computing unit on the host 100, it first needs to create a context on the corresponding heterogeneous computing unit. The context may include a pointer to the corresponding heterogeneous computing unit. The context is used to establish a connection between the application 810 and the heterogeneous computing unit. That is, a context of the application 810 uniquely corresponds to a heterogeneous computing unit. For example, Figure 8b According to some embodiments of the present application, a schematic diagram of establishing a connection between an application and a heterogeneous computing unit is shown.

[0139] like Figure 8b As shown, in the host 100, application A1 and application A2 are run, and the host 100 includes a heterogeneous computing unit B1 and a heterogeneous computing unit B2. When application A1 needs to use the heterogeneous computing unit B1, it needs to establish a context with the heterogeneous computing unit B1, for example, creating a context C1. When application A2 needs to use the heterogeneous computing unit B2, it needs to establish a context with the heterogeneous computing unit B2, for example, creating a context C2. Application A3 and application A4 are run in the host 100', and the host 100' includes a heterogeneous computing unit B3 and a heterogeneous computing unit B4. Application A3 is connected to the heterogeneous computing unit B4 by creating a context C3, and application A4 is connected to the heterogeneous computing unit B3 by creating a context C4.

[0140] In the distributed system of the present application, the application 810 exists in each host 100 in the distributed system, and a complex computing task can be divided into multiple applications 810 (ie, subtasks) and distributedly scheduled to be executed on each host 100 in the distributed system.

[0141] Below, based on Figure 8a Examples and Figure 8b The implementation introduces the data flow between heterogeneous computing units, which includes the creation and publishing process of heterogeneous objects and the distributed reading and writing process of heterogeneous objects.

[0142] Figure 9a According to some embodiments of the present application, an interactive flow chart of creation and publication of heterogeneous objects is shown.

[0143] like Figure 9aAs shown, the interaction process includes:

[0144] 901 , the application 810 applies for heterogeneous memory from the heterogeneous object management module 710 through the client module 820 .

[0145] For example, in some embodiments of the present application, when the application 810 needs to use a heterogeneous computing unit to perform a computing task or place data on a heterogeneous computing unit (i.e., to transfer data), the application 810 initiates a request for heterogeneous memory to the heterogeneous object management module 710 on the host 100 based on the interface (e.g., create) provided by the client module 710, thereby obtaining a piece of memory on a heterogeneous computing unit. The request carries the number (device id) of the heterogeneous computing unit where the context of the application 810 is located and the size (memorysize) of the required memory space (hereinafter referred to as the target memory space).

[0146] 902 , the heterogeneous object management module 710 allocates heterogeneous memory and returns the buffer to the application 810 .

[0147] Exemplarily, after receiving the request from the application 810, the heterogeneous object management module 710 first parses and obtains the number of the heterogeneous computing unit and the size of the target memory space carried in the request.

[0148] For example, the number of the heterogeneous computing unit is 001, and the target memory space size of the application is 100 bytes. The heterogeneous object management module 710 will allocate a continuous memory block of 100 bytes on the heterogeneous computing unit numbered 001, that is, the heterogeneous memory space corresponding to the flow data. Secondly, based on the cross-process shared memory mechanism of the heterogeneous computing unit, the heterogeneous object management module 710 sends the identification information (first storage identification) of the heterogeneous memory space (such as the CUDA memory handle of the GPU, the pointer, the address of the heterogeneous memory space, etc.) to the client module 820 of the application 810. The client module 820 parses the identification information of the heterogeneous memory space, determines the corresponding heterogeneous memory space, and returns the heterogeneous memory space to the application 810 in the form of a buffer for use by the application 810. Among them, the buffer is a data structure that encapsulates the pointer of the heterogeneous memory space, the space size and the number of the corresponding heterogeneous computing unit, and provides a read and write interface to the outside.

[0149] 903, the application 810 stores data in the heterogeneous memory space through the buffer, and sends a notification of releasing the buffer to the heterogeneous object management module.

[0150] Exemplarily, after the application 810 obtains the buffer returned by the heterogeneous object management module 710, it can operate the buffer based on the read and write interface provided by the buffer. For example, the intermediate results (i.e., flowable data) generated by the calculation process of the application 810 are saved in the heterogeneous memory space through the buffer, so as to facilitate the use of subsequent calculation steps. After the flowable data is saved, the application 810 can further call the publishing interface function buffer.Publish() provided by the buffer to inform the heterogeneous object management module 710 to publish the buffer.

[0151] 904, the heterogeneous object management module 710 maps the heterogeneous memory space corresponding to the buffer into a heterogeneous object.

[0152] Exemplarily, after receiving the notification of publishing the buffer, the heterogeneous object management module 710 maps the heterogeneous memory space corresponding to the buffer into a heterogeneous object, generates a UUID for the heterogeneous object, and returns the UUID to the client module 820 of the application 810 .

[0153] Exemplarily, in this embodiment, the heterogeneous memory mapping module 831 maps the heterogeneous memory space corresponding to the buffer into a heterogeneous object, and generates a UUID for the heterogeneous object. In addition, the heterogeneous memory mapping module 831 also saves the first mapping relationship between the UUID and the heterogeneous memory space. The heterogeneous memory mapping module 831 can obtain the heterogeneous memory space information of the heterogeneous object corresponding to the UUID based on the UUID and the first mapping relationship.

[0154] It should be understood that after the heterogeneous object management module 710 returns the UUID of the heterogeneous object to the client module 820, the applications of all hosts in the distributed system can obtain the corresponding UUID from the client module 820, and obtain the transferable data in the heterogeneous object corresponding to the UUID based on the UUID. The process of the application obtaining the transferable data of the heterogeneous object corresponding to the UUID is described below.

[0155] Next, the heterogeneous object management module 710 asynchronously saves the metadata of the heterogeneous object (eg, the mapping relationship between the UUID of the heterogeneous object and the identifier of the heterogeneous object management module 710 to which it belongs) to a metadata storage module of a heterogeneous object management module in the distributed system.

[0156] Exemplarily, the heterogeneous object management module 710 performs consistent hashing on the UUID of the heterogeneous object, and selects a heterogeneous object management module on a host in the distributed system according to the hash result (i.e., determines the address of the heterogeneous object) to save the metadata of the heterogeneous object corresponding to the UUID. At this point, the creation and release of the heterogeneous object is completed.

[0157] It should be understood that the heterogeneous object management module that stores the metadata of the heterogeneous object may also be the heterogeneous object management module 710 having the same address as the heterogeneous object.

[0158] During the creation and release of the above heterogeneous objects, the application 810 stores the transferable data in the heterogeneous memory space of the corresponding heterogeneous computing unit. Then the heterogeneous object management module 710 generates a globally unique UUID in the distributed system for the heterogeneous memory space, and returns the UUID to the application 810 through the client module 820. The application 810 can send the UUID to other applications in the distributed system based on the message passing in the distributed system. Therefore, the applications in all hosts in the distributed system can obtain the UUID and then obtain the transferable data in the heterogeneous object corresponding to the UUID.

[0159] Next, the distributed reading and writing process of the above heterogeneous objects is introduced.

[0160] Figure 9b According to some embodiments of the present application, an interactive flow chart of a heterogeneous object reading and writing process is shown.

[0161] like Figure 9b As shown, the process includes:

[0162] 905 , the application 810 sends the UUID corresponding to the target heterogeneous object to the heterogeneous object management module 710 .

[0163] For example, in some embodiments of the present application, if the application 810 wants to obtain the transferable data of a heterogeneous object, it can initiate a request (Get request) to the heterogeneous object management module 710 through the client module 820. The request sends the UUID of the heterogeneous object to the heterogeneous object management module 710.

[0164] 906 , the heterogeneous object management module 710 determines the address of the target heterogeneous object based on the UUID.

[0165] Exemplarily, after the heterogeneous object management module 710 (here assumed to be worker A) receives the request from the application 810 to obtain the target heterogeneous object, it parses and obtains the UUID information carried in the request, and hands it over to its own addressing module 834 according to the addressing rule (such as performing consistent hashing on the UUID, the consistent hashing rule is the same as the consistent hashing rule when storing the metadata of the heterogeneous object in the 904 process) to obtain the address of the metadata of the target heterogeneous object corresponding to the UUID (that is, the identifier of the heterogeneous object management module where the metadata is located, assuming it is worker B). The heterogeneous object management module 710 initiates a request to worker B to obtain the address of the target heterogeneous object corresponding to the UUID. After receiving the request, worker B obtains the metadata of the target heterogeneous object corresponding to the UUID (that is, the identifier of the worker where the target heterogeneous object corresponding to the UUID is located, assuming it is worker C) in its own metadata storage module 833, and returns the metadata information (worker C address) to worker A, thereby determining the address of the target heterogeneous object.

[0166] In other embodiments, the heterogeneous object management module 710 may also directly obtain the address of the target heterogeneous object corresponding to the UUID (ie, the UUID contains data on the address of the target heterogeneous object) according to the UUID, so as to determine the address of the target heterogeneous object.

[0167] It should be understood that in some embodiments of the present application, worker A, worker B, and worker C are heterogeneous object management modules on different hosts in a distributed system. In other embodiments, worker A, worker B, and worker C may also be heterogeneous object management modules on the same host.

[0168] 907 , the heterogeneous object management module 710 obtains information of the corresponding heterogeneous memory space based on the UUID of the target heterogeneous object and the address of the target heterogeneous object.

[0169] Exemplarily, after worker A obtains the address of the target heterogeneous object corresponding to the UUID (worker ID of worker C), worker A initiates a request to worker C to obtain the heterogeneous memory space information corresponding to the target heterogeneous object. After worker C receives the request, it obtains the information of the heterogeneous memory space corresponding to the target heterogeneous object (for example, the pointer and size of the heterogeneous memory space, and the encoding of the heterogeneous computing unit where it is located) through the heterogeneous memory mapping module 831 in worker C. Worker C sends the information of the heterogeneous memory space to worker A.

[0170] Exemplarily, the heterogeneous memory mapping module 831 of worker C may obtain the information of the heterogeneous memory space from the aforementioned first mapping relationship according to the UUID.

[0171] It should be understood that when worker C and worker A are not the same heterogeneous object management module (i.e., worker C and worker A are located in different hosts), worker C and worker A communicate via RPC. Worker C can send information about the heterogeneous memory space of the heterogeneous memory corresponding to the UUID to worker A via RPC communication.

[0172] 908 , the heterogeneous object management module 710 returns the transferable data in the heterogeneous memory space to the application 810 .

[0173] Exemplarily, after worker A obtains the information of the heterogeneous memory space corresponding to the UUID, it can obtain the transferable data in the heterogeneous memory space according to the information of the heterogeneous memory space, and return the transferable data to the application 810.

[0174] For example, worker A may return the flowable data in the heterogeneous memory space to the application 810 based on a pass-by-value method.

[0175] It should be understood that when worker A and worker C are two different heterogeneous object management modules 710, or when worker A and worker C are the same heterogeneous object management module 710, but the above-mentioned heterogeneous memory space cannot be shared across processes by the application 810, that is, memory sharing cannot be achieved between the heterogeneous computing unit (hereinafter referred to as the storage unit) where the heterogeneous memory space is located and the heterogeneous computing unit (hereinafter referred to as the target unit) corresponding to the context of the application 810, worker A can return a heterogeneous memory buffer to the client module 820 of the application 810 based on a pass-by-value method.

[0176] For example, worker A allocates a continuous memory space (hereinafter referred to as the target space) on the target unit, and copies the transferable data in the heterogeneous memory space where the target heterogeneous object corresponding to the UUID is located to the target space, and returns the target space in the form of a buffer to the application 810 for reading and writing. The application 810 can obtain the copied transferable data in the target space according to the buffer of the target space, thereby completing the data transfer of the heterogeneous computing unit.

[0177] It should be understood that when worker A and worker C are two different heterogeneous object management modules 710, the copy process can be copied across hosts via network transmission. When worker A and worker C are the same heterogeneous object management module 710, the copy process is a copy across heterogeneous computing units within the host.

[0178] It should be understood that in some other embodiments, worker A may also return the buffer of the heterogeneous memory space corresponding to the target heterogeneous object to the client module 820 of the application 810 based on a pass-by-reference method.

[0179] It should be understood that when worker A and worker C are the same heterogeneous object management module 710, and the heterogeneous memory space corresponding to the UUID can be shared across processes by the application 810. That is, the storage unit and the target unit run in a heterogeneous object management module 710, and the storage unit and the target unit can share memory across processes. Worker A can return the information of the heterogeneous memory space corresponding to the target heterogeneous object corresponding to the UUID to the client module 820. The client module 820 encapsulates the information of the heterogeneous memory space into a buffer and returns it to the application 810, and the application 810 obtains the flowable data in the heterogeneous memory space in a copy-free manner.

[0180] Exemplarily, the memory sharing (cross-process sharing) between the above-mentioned heterogeneous computing units may also be referred to as sharing based on a chip interconnect protocol in other embodiments. The chip interconnect protocol may, for example, include a high-speed peripheral component interconnect express (PCIe) interface, a compute express link (CXL) interface, or an NVLink communication protocol.

[0181] In some embodiments of the present application, the way in which the heterogeneous object management module 710 chooses to obtain the target heterogeneous object by passing by value or by reference is called topology-aware heterogeneous data flow. Because the heterogeneous object management module needs to determine the communication topology between the heterogeneous computing unit (target unit) where the context of the application 810 is located and the heterogeneous computing unit (storage unit) where the target heterogeneous object corresponding to a certain UUID that needs to be obtained is located. For example, the heterogeneous object management module 710 needs to determine whether the storage unit and the target unit are the same heterogeneous computing unit. If so, determine whether they support memory sharing, and determine whether the corresponding heterogeneous object management modules between the storage unit and the target unit support RDMA data transmission. The detailed process of the heterogeneous object management module 710 determining the relationship between the target unit and the storage unit is described below.

[0182] It should be understood that when the application 810 obtains the target heterogeneous object corresponding to a certain UUID, if the target heterogeneous object has not yet been generated, the heterogeneous object management module 710 can subscribe to the worker where the metadata of the target heterogeneous object corresponding to the UUID is stored, and block and wait (set a timeout). When the target heterogeneous object is generated and its metadata is published, the message that the heterogeneous object management module 710 needs to obtain the metadata of the target heterogeneous object corresponding to the UUID can be pushed according to the subscription relationship. (If the push fails within the timeout, a failure is returned).

[0183] Next, a process of data flow in a distributed system is introduced based on the above method.

[0184] Fig.10 According to some embodiments of the present application, an architecture diagram of a distributed system is shown.

[0185] Illustratively, in some embodiments of the present application, the distributed system includes a host 100 and a host 100'. The host 100 and the host 100' may be servers, for example.

[0186] like Fig.10 As shown, this embodiment takes the implementation of the host 100 as an example. A computing task (hereinafter referred to as the first computing task) can be divided into two main computing steps in the host 100: an application 811 and an application 812. Among them, the client module 820 includes a client toolkit 821 and a client toolkit 822. The application 811 interacts with the heterogeneous object management module 710 through the client toolkit 821, and the application 812 interacts with the heterogeneous object management module 710 through the client toolkit 822. The application connects to a specific heterogeneous computing unit by creating a context. In this embodiment, the application 811 creates a context connection to GPU0 through the cuCtxCreate interface provided by CUDA, and the application 812 uses the same method to connect to GPU1. The software components in the heterogeneous object management module 710 refer to Figure 8a Embodiment. The various component modules described in the heterogeneous object management module 710 are not described in detail here. In this embodiment, the heterogeneous object management module 710 in the host 100 can be called worker1.

[0187] In the hardware layer of the host 100, the components involved in this embodiment include CPU memory 850, GPU 861 and wireless broadband network card 860. Exemplarily, the network card 860 can be, for example, a wireless broadband network card, which provides efficient communication capabilities through the RDMA protocol. Exemplarily, GPU 861 is deployed on the host 100 in the form of a common single-machine eight-card, with four GPUs in a group. For example, the first group of GPUs includes GPU0 to GPU3, and the second group of GPUs includes GPU4 to GPU7. In each group of GPUs, a CPU is managed via a PCIe Switch (for example, CPU0 manages the first group of GPUs, and CPU2 manages the second group of GPUs), and CPU0 and CPU1 are connected via a quick path interconnect (QPI) protocol.

[0188] The implementation on the host 100' refers to the module on the host 100, which will not be repeated here. In some embodiments of the present application, the heterogeneous object management module 710 on the host 100' can be called worker2. The system described in this embodiment is distributedly deployed on two hosts to realize the heterogeneous data flow of distributed computing tasks. To facilitate the subsequent specific steps, the data flow diagram of the first computing task is shown as follows. Fig.11 As shown, the output of application 811 is the flowable data in the memory of GPU0, which needs to be passed to application 812 (shared in the same process) and application 811' (shared across processes) as input. Similarly, the output of application 811' is the flowable data in the memory of GPU2, which needs to be passed to application 812' as input (shared in the same process).

[0189] Below, based on Fig.10 The architecture of the distributed system shown introduces the data flow method between heterogeneous computing units proposed in this application.

[0190] First, the process of an application applying for heterogeneous memory space is introduced. Taking application 811 as an example, during the first computing task, the flowable data generated by application 811 needs to be passed to application 812 and application 811' as input. After generating the flowable data, application 811 needs to apply for heterogeneous memory space from the heterogeneous computing unit (i.e., GPU0) that has established a context with application 811 to store the flowable data.

[0191] Fig.12 According to some embodiments of the present application, a flowchart of an implementation of an application applying for heterogeneous memory space is shown.

[0192] like Fig.12 As shown, the process includes:

[0193] 1201 , the application 811 applies for heterogeneous memory space from the heterogeneous object management module 710 through the client toolkit 821 .

[0194] For example, in some embodiments of the present application, the application 811 needs to apply for memory space, and the client toolkit 821 can obtain the context created by the application 811 and the heterogeneous computing unit. For example, the application 811 is connected to GPU0 through the context. Then the client toolkit 821 sends a request to the heterogeneous object management module 710 to apply for the memory space of GPU0 (i.e., the heterogeneous memory space).

[0195] It should be understood that in other embodiments, the application 811 may establish a context with other heterogeneous computing units, such as GPU1, and the heterogeneous computing unit for which the application 811 establishes the context is not limited.

[0196] 1202, the heterogeneous object management module 710 allocates the memory space requested by the application 811 on GPU0, and determines whether the memory space is allocated successfully.

[0197] Exemplarily, after the heterogeneous object management module 710 obtains the request of the application 811, it can allocate corresponding memory space on the memory of GPU0 and determine whether the memory allocation is successful.

[0198] If the judgment result is yes, that is, the heterogeneous object management module 710 successfully allocates memory space on GPU0, the process 1204 is executed, and the heterogeneous object management module 710 obtains the CUDA IPC memory handle corresponding to the allocated memory space and returns it to the client toolkit 821.

[0199] If the judgment result is no, that is, the heterogeneous object management module 710 fails to allocate memory on GPU0, the process 1203 is executed, and the heterogeneous object management module 710 returns a message of memory space application failure to the client toolkit 821.

[0200] Exemplarily, when the free memory space on GPU0 is smaller than the heterogeneous memory space requested by the application 811, or GPU0 fails, the heterogeneous object management module 710 will fail to allocate memory space to GPU0.

[0201] It should be understood that the allocation of heterogeneous memory space is implemented by the heterogeneous memory mapping module 831 of the heterogeneous object management module 710 , and the communication management module 832 may be responsible for receiving and processing the request of the client toolkit 821 by the heterogeneous object management module 710 .

[0202] 1203 , the heterogeneous object management module 710 returns a message indicating that the memory space application has failed to the client toolkit 821 .

[0203] Exemplarily, when the heterogeneous object management module 710 fails to allocate memory space to GPU0, it returns a message of memory space application failure to the client toolkit 821, for example, an error code may be sent to the client toolkit 821. Then, the process of the application 811 applying for heterogeneous memory space ends.

[0204] 1204 , the heterogeneous object management module 710 obtains the CUDA IPC memory handle corresponding to the allocated memory space, and returns it to the client toolkit 821 .

[0205] Exemplarily, after the heterogeneous object management module 710 successfully allocates memory to GPU0, the identifier of the allocated memory can be sent to the client toolkit 821. For example, the heterogeneous object management module 710 allocates memory space to the heterogeneous computing unit through the management toolkit corresponding to the heterogeneous computing unit, determines the CUDA IPC memory handle corresponding to the allocated memory space based on the IPC mechanism of CUDA, and then returns the CUDA IPC memory handle of the allocated memory to the client toolkit 821.

[0206] 1205 , the client toolkit 821 parses the CUDA IPC memory handle, obtains the memory space allocated by the heterogeneous object management module 710 and encapsulates it into a buffer structure.

[0207] Exemplarily, the client toolkit 821 may open the CUDA IPC memory handle to obtain the information of the heterogeneous memory space, and encapsulate the heterogeneous memory space information into a buffer.

[0208] It should be understood that when the heterogeneous memory space is used by other applications in the distributed system, the heterogeneous memory space no longer needs to be used for data flow, and the entire computing task no longer needs to use the heterogeneous memory space. The buffer can also automatically close the CUDA IPC memory handle, thereby releasing the corresponding heterogeneous memory space normally to clean up the memory space of the heterogeneous computing unit, so that other applications can store transferable data in the corresponding heterogeneous computing unit to facilitate data flow in the distributed system.

[0209] 1206 , the client toolkit 821 returns the buffer to the application 811 .

[0210] Exemplarily, after the client toolkit 821 encapsulates the heterogeneous memory space into a buffer, it sends the buffer to the application 811. The application 811 can save the flowable data generated by its own computing task in the applied heterogeneous memory space through the buffer for subsequent data flow.

[0211] In the process of the above application 811 applying for heterogeneous memory space, since the heterogeneous computing units in the host 100 are all managed by the heterogeneous object management module 710, the application 811 only needs to send the message of the memory of the required heterogeneous computing unit to the heterogeneous object management module 710 as needed. After obtaining the applied heterogeneous memory space, the application 811 can store the data generated by itself or the function to be calculated in the heterogeneous memory space based on the buffer for subsequent release.

[0212] The following describes the process of encapsulating heterogeneous memory spaces into heterogeneous objects and publishing them.

[0213] Fig.13 According to some embodiments of the present application, a flowchart for implementing heterogeneous object publishing is shown.

[0214] like Fig.13 As shown, the process includes:

[0215] 1207 , the application 811 notifies the heterogeneous object management module 710 to publish the buffer through the client toolkit 821 .

[0216] Exemplarily, the application 811 may call the publishing interface function buffer.Publish() provided by the buffer to notify the heterogeneous object management module 710 to publish the buffer encapsulated by the client toolkit.

[0217] 1208 , the heterogeneous object management module 710 maps the heterogeneous memory space corresponding to the buffer into a heterogeneous object, and returns the UUID of the heterogeneous object to the client toolkit 821 .

[0218] Exemplarily, after the heterogeneous object management module 710 receives a message to publish the buffer from the client toolkit 821, it can map the heterogeneous memory space corresponding to the buffer into a heterogeneous object. For example, after the communication management module 832 in the heterogeneous object management module 710 receives the publishing request from the client toolkit 821, it sends the request to the heterogeneous memory mapping module 831, and the heterogeneous memory space mapping module 831 maps the heterogeneous memory space corresponding to the buffer into a heterogeneous object (the heterogeneous memory mapping module 831 maintains a hash table to record the mapping relationship), and assigns a globally unique UUID to the heterogeneous object, and immediately returns the UUID to the client toolkit 821 in a non-blocking manner (without setting a waiting time) for the application 811 to pass the UUID to other applications in the distributed system.

[0219] It should be understood that after allocating UUIDs for heterogeneous objects, the heterogeneous memory mapping module 831 will also store the first mapping relationship between the UUIDs and the heterogeneous objects, so that the heterogeneous memory mapping module 831 can determine the information of the heterogeneous objects based on the UUIDs.

[0220] 1209, the heterogeneous object management module 710 saves the metadata of the heterogeneous object to a certain worker.

[0221] Exemplarily, after the heterogeneous memory mapping module 831 assigns UUID to the heterogeneous object, the heterogeneous object metadata module 833 asynchronously stores the mapping relationship between the UUID and the address of the heterogeneous object management module 710 (i.e., the metadata of the heterogeneous object) in the metadata storage module 833 of a worker in the distributed system.

[0222] Exemplarily, the storage address of the metadata of the heterogeneous object can be obtained by performing consistent hashing on the UUID by the heterogeneous object metadata module 833. The storage address of the metadata can be the heterogeneous object management module 710 (worker1) or a worker in another host in the distributed system.

[0223] During the publishing process of the above heterogeneous objects, the heterogeneous object management module 710 assigns a UUID to the heterogeneous objects. The UUID is globally visible in the distributed system, that is, other applications in the distributed system can obtain the UUID. Therefore, in the distributed system, when the data of the heterogeneous computing unit is transferred, the application 811 can determine the transferable data to be exchanged through the UUID, and then obtain the transferable data of the heterogeneous object corresponding to the UUID.

[0224] It should be understood that Fig.11 In the embodiment of the present invention, the output of application 811' needs to be passed to application 812', so application 811' also needs to publish its own data. The publishing process is similar to Fig.12 and Fig.13 The process shown is the same and will not be repeated here.

[0225] The following describes the process of an application obtaining heterogeneous objects.

[0226] Fig.14 According to some embodiments of the present application, a flowchart of an implementation of an application acquiring heterogeneous objects is shown.

[0227] 1401 , the application sends a request for obtaining data of a heterogeneous object corresponding to a UUID to the heterogeneous object management module 710 through the client module 820 .

[0228] Exemplarily, after the application 811 publishes its corresponding heterogeneous object, the application 811 can send the UUID to the application 812 and the application 811' in a message passing manner. When the application 812 obtains the UUID, it can initiate a request (for example: buffer = Get (UUID), the request carries the context information of the application 812) to the heterogeneous object management module 710 (worker1) corresponding to the application 812 through the interface provided by the client toolkit 822 to obtain the flowable data corresponding to the heterogeneous object corresponding to the UUID.

[0229] It should be understood that the application 811' will also initiate a request to the heterogeneous object management module 710 (worker2) corresponding to the application 811' through the interface provided by the client toolkit 821' to obtain the transferable data corresponding to the heterogeneous object corresponding to the UUID.

[0230] 1402, the heterogeneous object management module 710 obtains the address of the heterogeneous object corresponding to the UUID based on the UUID.

[0231] Exemplarily, after receiving the request, the heterogeneous object management module 710 first performs consistent hashing on the UUID by its addressing module 834 to determine the identifier of the heterogeneous object management module 710 that stores the metadata of the heterogeneous object.

[0232] Exemplarily, in this embodiment, the heterogeneous object management module 710 corresponding to the application 812 is worker1, and the address of the metadata of the heterogeneous object may be, for example, worker3 (worker3 and worker1 may also be the same worker).

[0233] Then, the metadata storage module 833 of worker3 can obtain the address of the heterogeneous object according to the metadata of the heterogeneous object, for example, worker1 (the UUID is published by application 811, and the address of the heterogeneous object corresponding to application 811 is worker1).

[0234] In other embodiments, when the application 811' obtains data of a heterogeneous object, the worker 2 corresponding to the application 811' may also obtain the address of the heterogeneous object corresponding to the UUID through the above method.

[0235] 1403, the heterogeneous object management module 710 determines a storage unit and a target unit.

[0236] Exemplarily, after the address of the heterogeneous object corresponding to the UUID is determined, the heterogeneous computing unit where the heterogeneous object is located can be determined.

[0237] Exemplarily, in this embodiment, the heterogeneous object is in worker 1, so the heterogeneous memory mapping module 831 of worker 1 further queries and obtains the heterogeneous computing unit (ie, storage unit) where the heterogeneous object is located.

[0238] Exemplarily, in this embodiment, the heterogeneous memory mapping module 831 can determine the heterogeneous computing unit corresponding to the heterogeneous object according to the UUID and the first mapping relationship, for example, the heterogeneous computing unit is GPU0.

[0239] It should be understood that when determining the target unit, the heterogeneous object management module 710 (i.e., worker 1) corresponding to the application 812 can determine the heterogeneous computing unit that establishes a context with the application 812 based on the context information of the application 812 based on the communication management module 831, that is, determine the target unit. In this embodiment, the target unit is GPU1 on the host 100.

[0240] 1404, the heterogeneous object management module 710 obtains the communication topology between the storage unit and the target unit.

[0241] Exemplarily, after determining the storage unit and the target unit, the heterogeneous object management module 710 can select pass by value or pass by reference to obtain the data of the heterogeneous object. The method of selecting the method of obtaining the data of the heterogeneous object is to obtain the communication topology of the storage unit and the target unit.

[0242] 1405 , the heterogeneous object management module 710 determines whether the storage unit and the target unit are located in the same node.

[0243] It should be understood that the storage unit and the target unit being located at the same node means that the target unit and the storage unit are located at the same host.

[0244] For example, whether the target unit and the storage unit are located in the same host can be determined by whether the identifiers of the heterogeneous object management module 710 are the same. The mapping relationship between the host where the heterogeneous computing unit belongs to the heterogeneous object management module 710 can be determined in the communication management module 832 of the heterogeneous object management module 710.

[0245] If the judgment result is yes, that is, the storage unit and the target unit are located in the same node, the process 1406 is executed, and the heterogeneous object management module 710 determines whether the storage unit and the target unit support P2P access.

[0246] If the judgment result is no, that is, the storage unit and the target unit are not located in the same node, the process 1409 is executed, and the heterogeneous object management module 710 allocates memory in the target unit and copies the data of the heterogeneous object to the memory.

[0247] Exemplarily, in this embodiment, GPU0 and GPU1 are both located in the host 100 and belong to the same node, and GPU0' belongs to the host 100', so GPU0 and GPU2' do not belong to the same node.

[0248] 1406 , the heterogeneous object management module 710 determines whether the storage unit and the target unit support P2P access.

[0249] Exemplarily, if the judgment result is yes, that is, the storage unit and the target unit support P2P access, then 1407 is executed, and the heterogeneous object management module 710 returns the information of the heterogeneous memory space to the client module 820 in a reference transfer manner.

[0250] If the judgment result is no, that is, the storage unit and the target unit do not support P2P access, then 1409 is executed, and the heterogeneous object management module 710 allocates memory in the target unit and copies the data of the heterogeneous object to the memory.

[0251] 1407 , the heterogeneous object management module 710 returns the information of the heterogeneous memory space to the client module 820 in a reference passing manner.

[0252] Exemplarily, the storage unit and the target unit are located on the same node and support P2P access, then the communication management module 831 of the heterogeneous object management module 710 can send information about the heterogeneous memory space where the heterogeneous object corresponding to the UUID is located, such as identification information of the heterogeneous memory space (i.e., the CUDA IPC memory handle), to the client module 820.

[0253] Exemplarily, in this embodiment, the client module 820 corresponding to the application 812 is a client toolkit 822 , that is, the communication management module 831 of worker 1 sends the CUDA IPC memory handle of the corresponding heterogeneous memory space to the client toolkit 822 .

[0254] It should be understood that in other embodiments, the client module 820 corresponding to the application 811' is a client toolkit 821, but because the heterogeneous computing unit GPU2' and GPU0 to which the application 811' context is connected are not the same node, the application 811' cannot obtain the data of the heterogeneous object by passing by reference.

[0255] 1408. The client module 820 encapsulates the heterogeneous memory space into a buffer for the application to use.

[0256] Exemplarily, in some embodiments, the client module 820 is a client toolkit 822. The client toolkit 822 obtains the identification information of the heterogeneous memory space where the heterogeneous object is located, that is, the CUDA IPC memory handle. The client toolkit 822 can directly open the memory handle, obtain the information of the heterogeneous memory space corresponding to the heterogeneous object without copying, and encapsulate the information of the heterogeneous memory space into a buffer and send it to the application 812 for use. At this time, the application 812 obtains the transferable data published by the application 811 from the heterogeneous memory space based on the buffer to complete the data transfer.

[0257] For example, in some other embodiments, the client module is a client toolkit 821', which encapsulates the information of the heterogeneous memory space (target memory space in this embodiment) of GPU2' storing the transferable data into a buffer, and then sends the buffer to the application 811'. The application 811' performs read and write operations on the transferable data in the target memory space in GPU2' according to the buffer, thereby completing the data transfer between GPU0 and GPU2'.

[0258] 1409 , the heterogeneous object management module 710 allocates a target memory space in the target unit.

[0259] Exemplarily, when the storage unit and the target unit do not belong to the same node or do not support P2P access, the storage unit and the target unit need to exchange data in a copying manner. The heterogeneous object management module 710 will allocate corresponding target memory space in the target unit to receive the data of the heterogeneous object, and the size of the allocated target memory space can be directly obtained during the heterogeneous object addressing process.

[0260] Exemplarily, when application 811' obtains the flowable data of the heterogeneous object corresponding to the UUID published by application 811, since GPU2' to which the context of application 811' is connected and the heterogeneous computing unit GPU0 corresponding to the heterogeneous object are not the same node, the heterogeneous object management module 710 (i.e., worker2) corresponding to application 811' allocates target memory space on GPU2' to obtain the flowable data in the heterogeneous object.

[0261] 1410, the heterogeneous object management module 710 determines whether the storage unit and the target unit support RDMA.

[0262] Exemplarily, if the judgment result is yes, that is, the storage unit and the target unit support RDMA, 1411 is executed, and the heterogeneous object management module 710 copies the transferable data in the heterogeneous memory space based on RDMA transmission.

[0263] If the judgment result is no, that is, the storage unit and the target unit do not support RDMA, then 1413 is executed, and the heterogeneous object management module 710 copies the transferable data in the heterogeneous memory space through the CPU.

[0264] 1411, the heterogeneous object management module 710 copies the transferable data in the heterogeneous memory space based on RDMA transmission.

[0265] Exemplarily, the communication management module 831 of worker2 carries the network card 870' information of the host 100' and initiates a request to copy the information of the heterogeneous memory space to the communication management module 831 of worker1. After receiving the request, the communication management module 831 of worker1 matches the information of the network card 870 of the host 100 and finds that the RDMA transmission protocol is supported between the two. Therefore, worker2 copies the transferable data in the heterogeneous memory space corresponding to the heterogeneous object of GPU0 directly to the target memory space of the target unit (GPU2') through RDMA.

[0266] 1412 , the heterogeneous object management module 710 returns the transferable data in the heterogeneous memory space of the target unit to the client module 820 .

[0267] Exemplarily, the heterogeneous object management module 710 returns the target memory space information of the target unit to the client module 820 , and then 1408 is executed, where the client module 820 encapsulates the target memory space information into a buffer for the application 811 to use.

[0268] Exemplarily, in this embodiment, after worker2 copies the transferable data of GPU0 to the target memory space of GPU2', it can return the information of the target memory space corresponding to GPU2' storing the transferable data to the client toolkit 821', so that the subsequent application 811' can obtain the transferable data.

[0269] 1413, the heterogeneous object management module 710 copies the transferable data in the heterogeneous memory space through the CPU.

[0270] Exemplarily, if worker1 and worker2 find that cross-node RDMA is not supported during communication, the flowable data in the heterogeneous memory space can be transferred through the CPU memory, for example, through TCP or IP transmission, that is, data is passed by value. After Worker2 stores the flowable data in the target memory space, 1412 is executed, and the heterogeneous object management module 710 returns the flowable data in the heterogeneous memory space of the target unit to the client module 820.

[0271] For example, the scheme of transferring data through CPU memory is as follows Figure 4b Embodiment of the invention.

[0272] It should be understood that in other embodiments, the application 811' also publishes its output data as a heterogeneous object (the corresponding memory is on GPU2' of the host 100'), and the application 812' calls the UUID corresponding to the heterogeneous object published by the application 811' to obtain the flowable data of the heterogeneous object published by the application 811'. Then worker2 will look for the identifier of the worker corresponding to the heterogeneous object according to the UUID. The addressing process has been introduced above and will not be repeated here. After obtaining the identifier of the worker corresponding to the heterogeneous object, the heterogeneous object is stored on worker2 of the host 100'. The heterogeneous memory mapping module 831 in Worker2 determines that the heterogeneous object is stored on GPU2' of worker2 based on the UUID and the first mapping relationship. The context of the application 812' of the heterogeneous object management module 710' is GPU4', and it is determined that they are in the same node, but belong to different groups (GPU2' belongs to the group managed by CPU0': {GPU0'~GPU3'}, and GPU4' belongs to the group managed by CPU1: {GPU4'~GPU7'}). Since CUDA's P2P access cannot cross the QPI protocol, memory sharing cannot be achieved between GPU2' and GPU4'. Therefore, worker2's heterogeneous memory space mapping module 831 allocates heterogeneous memory space on GPU4', and copies the transferable data corresponding to the heterogeneous objects from GPU2' to the heterogeneous memory space of GPU4'. Then, worker2's communication management module 831 returns the information of the copied memory space on GPU4' directly to the client toolkit 822'. The client toolkit 822' encapsulates the information of the memory space into a buffer for the application 812' to use. The application 812' obtains the transferable data in the memory space of GPU4' according to the buffer. This completes the data flow between GPU2' and GPU4'.

[0273] It should be understood that the above-mentioned publishing of heterogeneous objects is an asynchronous, non-blocking (no waiting time is set) method, the purpose of which is to immediately return the UUID to the caller. The caller obtains the heterogeneous object in a blocking waiting manner (setting a waiting time). If the object is not obtained after the timeout, a message indicating that the UUID has failed to be obtained is returned.

[0274] In the data flow of the above heterogeneous computing unit keys, the heterogeneous object is defined as a distributed, globally unique data object. Heterogeneous objects are asynchronously published without waiting for all publishing actions to be completed, and the UUID is immediately returned to the calling application. The application can dynamically obtain the data of the heterogeneous object based on the UUID. The way to obtain the value of the heterogeneous object provides two semantics: addressing by value and addressing by reference, which can meet the needs of different scenarios.

[0275] The embodiment of the present application also provides a computing device 00, which can be used to implement one or more hosts 100. For example, the computing device 00 is a physical machine such as a server, a mobile phone, or a computer. That is, the computing device 00 alone serves as a host 100 in a distributed system. In other embodiments, the host 100 may also be a virtual machine in the operating system of the computing device 00. For example, the computing device 00 is a physical machine such as a server, a mobile phone, or a computer, and the host 100 is a virtual machine in the operating system of the computing device 00, that is, multiple virtual machines are run in the operating system of the computing device 00 as multiple hosts 100 in the distributed system.

[0276] Fig.15 According to some embodiments of the present application, a schematic structural diagram of a host is shown.

[0277] Exemplarily, the host 100 may be, for example, a computing device 00 .

[0278] like Fig.15 As shown, the host 100 includes at least one general-purpose processor 101 , at least one memory 102 , at least one heterogeneous computing unit 103 , and a bus 104 .

[0279] The general processor 101 may be a CPU, configured to execute instructions corresponding to the aforementioned application programs 810 , 811 , 812 , 811 ′, and 812 ′, so as to implement the data processing methods provided in the embodiments of the present application.

[0280] It should be understood that each general purpose processor 101 may be connected to at least one memory 102 for storing execution and data during the operation of the host 100. In some embodiments, the memory 102 may be a volatile memory, such as a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), a direct rambus RAM (DR RAM), etc.

[0281] The heterogeneous computing unit 103 may include one or more of a GPU, a TPU, an NPU, and an AI chip, for executing instructions corresponding to the aforementioned application 810, application 811, application 812, application 811', and application 812' to implement the data processing method provided in each embodiment of the present application.

[0282] It should be understood that in some embodiments, each heterogeneous computing unit 103 is connected to at least one heterogeneous computing unit memory 1031, and the heterogeneous computing unit memory 1031 may include a volatile memory, such as RAM, SRAM, DRAM, SDRAM, DDRSDRAM, ESDRAM, SLDRAM, DR RAM, high bandwidth memory (HBM), etc.

[0283] It should be understood that in some embodiments, the heterogeneous computing unit memory 1031 may also be set inside the heterogeneous computing unit 103.

[0284] The bus 104 may include the aforementioned PCIe bus, CXL bus, QPI bus, etc.

[0285] It should be understood that the bus 104 may also include other buses for communication between modules in the host 100, buses for communication between the host 100 and input / output devices, buses for communication between the host 100 and other devices (eg, other hosts), and the like.

[0286] Optionally, in some embodiments, the host 100 may further include a non-volatile memory for storing instructions and data involved in the methods provided in the embodiments of the present application.

[0287] Optionally, in some embodiments, the host 100 may further include a communication interface for communicating with other electronic devices (eg, other hosts), such as a network interface, TCP protocol, IP protocol, RDMA protocol, etc.

[0288] It should be understood that Fig.15 The structure of the host 100 shown is only an example. In other embodiments, the host 100 may also include more or fewer modules, which is not limited here.

[0289] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0290] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0291] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0292] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, a floppy disk, an optical disk, an optical disk, a read-only memory, a magneto-optical disk, a random access memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0293] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0294] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0295] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0296] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A data processing method, applied in a computing system, characterized in that: The computing system includes a first host and a second host, the first host includes a first heterogeneous computing unit, and the second host includes a second heterogeneous computing unit; The method comprises: Get the data to be processed; Allocating the processing task of the data to be processed to the first host and the second host, wherein the first intermediate data generated by the first heterogeneous computing unit of the first host corresponding to the data to be processed is the intermediate data required for the second heterogeneous computing unit of the second host to complete the processing task corresponding to the data to be processed; The allocating the processing task of the to-be-processed data to the first host and the second host includes: The first intermediate data generated by the first heterogeneous computing unit corresponding to the data to be processed has a first mapping relationship and a second mapping relationship, The first mapping relationship is a mapping relationship between a first storage space storing the first intermediate data on the first host and an identifier of the first intermediate data. The second mapping relationship is a mapping relationship between the identifier of the first intermediate data and the first host; The second heterogeneous computing unit of the second host obtains the first intermediate data from the first storage space in the first host based on the first mapping relationship and the second mapping relationship, and completes a processing task corresponding to the data to be processed based on the first intermediate data.

2. The method according to claim 1, characterized in that The first storage space is a storage space in the memory of the first heterogeneous computing unit.

3. The method according to claim 2, characterized in that The second heterogeneous computing unit of the second host obtains first intermediate data from the first storage space in the first host based on the first mapping relationship and the second mapping relationship, including: The second host obtains the second mapping relationship from the host of the computing system based on the identifier of the first intermediate data; The second host obtains, from the first host, a first storage identifier of the first storage space based on the second mapping relationship; The first storage identifier includes a pointer to the first storage space and / or an address of the first storage space; The second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier.

4. The method according to claim 3, characterized in that The second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, including: Corresponding to the chip interconnection protocol being supported between the first heterogeneous computing unit and the second heterogeneous computing unit, the second host sending the first storage identifier to the second heterogeneous computing unit; The second heterogeneous computing unit obtains the first intermediate data from the first storage space based on the chip interconnection protocol and the first storage identifier.

5. The method according to claim 3, characterized in that: The second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, further comprising: Corresponding to the first heterogeneous computing unit and the second heterogeneous computing unit not supporting a chip interconnection protocol and the first heterogeneous computing unit and the second heterogeneous computing unit supporting remote direct memory access; The second host allocates a second storage space in the second heterogeneous computing unit; The second heterogeneous computing unit obtains the first intermediate data from the first heterogeneous computing unit through remote direct memory access based on the first storage identifier, and stores the first intermediate data in the second storage space.

6. The method according to claim 3, characterized in that The first host further includes a first central processor and a first memory corresponding to the first central processor, and the second host further includes a second central processor and a second memory corresponding to the second central processor; and the second host transmits the first intermediate data to the second heterogeneous computing unit based on the first storage identifier, further comprising: Corresponding to the chip interconnection protocol not being supported between the first heterogeneous computing unit and the second heterogeneous computing unit, The first heterogeneous computing unit and the second heterogeneous computing unit do not support remote direct memory access; The second host allocates a second storage space in the second heterogeneous computing unit; The first central processor stores the first intermediate data in the first memory based on the first storage identifier; The first central processor transmits the first intermediate data in the first memory to the second memory via a network; The second central processing unit transfers the first intermediate data from the second memory space to the second storage space.

7. The method according to any one of claims 4 to 6, characterized in that: The chip interconnect protocol includes any one of the following protocols: a high-speed peripheral component interconnect interface, a compute express link interface, or an NVLink communication protocol.

8. A computing system, characterized in that: The computing system includes a first host and a second host, the first host includes a first heterogeneous computing unit, and the second host includes a second heterogeneous computing unit; The first host and the second host are used to execute the processing task of the to-be-processed data acquired by the computing system, wherein: The first intermediate data generated by the first heterogeneous computing unit of the first host corresponding to the data to be processed is the intermediate data required for the second heterogeneous computing unit of the second host to complete the processing task corresponding to the data to be processed; The first host and the second host are used to execute the processing task of the to-be-processed data acquired by the computing system, including: The first host is used to set the first intermediate data generated by the first heterogeneous computing unit corresponding to the data to be processed to have a first mapping relationship and a second mapping relationship, The first mapping relationship is a mapping relationship between a first storage space storing the first intermediate data on the first host and an identifier of the first intermediate data. The second mapping relationship is a mapping relationship between the identifier of the first intermediate data and the first host; The second host is used to obtain the first intermediate data from the first storage space in the first host based on the first mapping relationship and the second mapping relationship through the second heterogeneous computing unit, and complete the processing task corresponding to the data to be processed based on the first intermediate data.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing instructions; At least one processor is used to execute the instructions so that the electronic device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the data processing method according to any one of claims 1 to 7.

Images