A third-party library calling method and device based on a distributed system

Abstract

The application provides a third-party library calling method and device based on a distributed system, the distributed system comprising at least one host and a plurality of devices; the method is applied to the host and comprises: determining a third-party distributed computing task; wherein the third-party distributed computing task is executed by a third-party library operator, a computing object of the third-party distributed computing task is first tensor data, and the first tensor data corresponds to first tensor information; second tensor information is converted according to the first tensor information, the second tensor information is expressed in a second data structure supported by the third-party library, and the first tensor information and the second tensor information are converted by taking a distributed tensor memory expressed data structure as a conversion medium; the third-party library operator sends computing instructions to the plurality of devices respectively according to the second tensor information, and the computing instructions contain the third-party distributed computing task. The method realizes the calling of an AI framework to a third-party library in the scenario of a distributed system.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method and apparatus for calling third-party libraries based on a distributed system. Background Technology

[0002] AI (Artificial Intelligence) frameworks are a set of standard interfaces, feature libraries, and toolkits for designing, training, and validating AI algorithm models. Developing and maintaining an operator within an AI framework is time-consuming. With the explosive growth of AI applications, users have increasingly diverse and customized requirements for operators. Developing operators from scratch requires significant time. Therefore, some frameworks, such as Mindspore and the JAX framework, provide solutions for integrating third-party libraries (such as Python libraries). These allow the operator capabilities of third-party libraries to be rearranged and combined into a new operator, which can then be called within the AI ​​framework. Because AI frameworks and third-party libraries use different tensor data structures, data type conversion is required when calling third-party libraries.

[0003] Currently, solutions for AI frameworks to call third-party libraries only consider single-card scenarios. In distributed scenarios, data conversion leads to repeated data transfer between the host and device sides, resulting in wasted computing performance. As high-performance computing applications become more common, the scenarios where AI frameworks call third-party distributed libraries (such as cuNumeric) to perform distributed computing are also increasing. There is an urgent need for an optimized method for AI frameworks to access third-party libraries in a distributed system environment. Summary of the Invention

[0004] This application provides a method and apparatus for calling third-party libraries based on a distributed system, which enables AI frameworks to call third-party libraries in distributed system scenarios without the need for data transfer.

[0005] Firstly, this application provides a method for calling a third-party library based on a distributed system, the distributed system including at least one host and multiple devices; the method is applied to the host in the distributed system and includes: determining a third-party distributed computing task; wherein the third-party distributed computing task is executed by a third-party library operator, the computing object of the third-party distributed computing task is first tensor data, the first tensor data corresponds to first tensor information; the first tensor information includes metadata and first distribution information of the first tensor data, the first distribution information representing the distribution information of the first tensor data distributedly stored on multiple devices in a first data structure supported by a deep learning framework; based on the first tensor information, a second tensor information is obtained, the second tensor information is represented in a second data structure supported by the third-party library, and the first tensor information and the second tensor information are converted using a distributed tensor memory representation data structure as the conversion medium; the third-party library operator sends computing instructions to multiple devices respectively based on the second tensor information, the computing instructions including the third-party distributed computing task, so that the multiple devices use their stored first tensor data to execute the third-party distributed computing task, and the multiple devices output the second tensor data after executing the third-party distributed computing task, the second tensor data being stored in a second data structure.

[0006] In this way, when the next task to be executed is a third-party distributed computing task, the host in the distributed system can collect the first tensor information of the first tensor data output by the operators of the deep learning framework. This first tensor information contains the first distribution information of the first tensor data distributed across multiple devices and the metadata of the first tensor data. A distributed tensor memory representation data structure is used as an intermediate data structure to transform it into the second tensor information. This realizes the transfer of the first distribution information and metadata from multiple devices between the deep learning framework and the third-party library. This allows the third-party library to continue to execute the distributed computing of the next operator using the first tensor data stored on each device based on the first distribution information and metadata of each device. This realizes the AI ​​framework's call to the third-party library in the distributed system scenario. Moreover, since only the distribution information needs to be transferred between different frameworks during the data type conversion process, the transfer of the first tensor data between the host and the devices can be avoided, thus avoiding the problem of performance waste.

[0007] In one possible implementation, the method further includes: in response to the output of second tensor data after multiple devices perform a third-party distributed computing task, obtaining third tensor information, the third tensor information including: metadata of the second tensor data and second distribution information, the second distribution information representing the distribution information of the second tensor data distributed across multiple devices in a second data structure; and converting the third tensor information to obtain fourth tensor information, the fourth tensor information represented by a distributed tensor memory representation data structure, the fourth tensor information being used by the deep learning framework to continue distributed computing.

[0008] In this way, after the third-party library operator finishes executing and outputs the second tensor data, the host in the distributed system can collect the second distribution information and metadata of the second tensor data, which are distributed across multiple devices. It can then encapsulate the second distribution information into a distributed tensor memory representation data structure. Using this distributed tensor memory representation data structure as an intermediate data structure, the second distribution information can be transferred between the third-party library and the deep learning framework. This allows the deep learning framework to use the second tensor data stored on each device, based on the second distribution information and metadata of the second tensor data, to continue performing subsequent distributed computations. Thus, in a distributed system scenario, after the AI ​​framework calls the third-party library, it can use the library's output to continue computation. Furthermore, since only distribution information needs to be transferred between different frameworks during data type conversion, the transfer of the second tensor data between the host and devices can be avoided, thus preventing performance waste.

[0009] In one possible implementation, after converting the fourth tensor information based on the third tensor information, the method further includes: sending a data rearrangement request to multiple devices, the data rearrangement request including the fourth tensor information and graph inference arrangement information, so that the devices rearrange the second tensor data after receiving the data rearrangement request, so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

[0010] In this way, the second tensor data output after the third-party library operator is calculated is rearranged to ensure that the output of the third-party library operator is consistent with the result of graph reasoning, so as to facilitate the calculation of the next operator.

[0011] In one possible implementation, the first tensor information also includes the stream information corresponding to the device.

[0012] In this way, adding stream information to the distributed tensor memory representation data structure allows stream information to be passed between two different frameworks. This enables the called third-party library operators to use the stream created by the deep learning framework when performing distributed computing, without having to create and maintain a new stream, reducing the overhead of stream creation and destruction, and saving the execution time of the operators.

[0013] In one possible implementation, the method further includes: obtaining user operation information; and setting stream information based on the user operation information.

[0014] In this way, users can directly manipulate the stream in the distributed tensor memory representation data structure to perform synchronous and asynchronous operations between the two frameworks.

[0015] Secondly, this application provides a third-party library invocation device based on a distributed system, the distributed system including at least one host and multiple devices; the device is applied to the host in the distributed system and includes: a determination module, used to determine a third-party distributed computing task; wherein the third-party distributed computing task is executed by a third-party library operator, the computing object of the third-party distributed computing task is first tensor data, the first tensor data corresponds to first tensor information; the first tensor information includes the metadata of the first tensor data and first distribution information, the first distribution information representing the distribution information of the first tensor data distributedly stored in multiple devices using a first data structure supported by a deep learning framework; an encapsulation module, used to convert the first tensor information to obtain second tensor information, the second tensor information is represented by a second data structure supported by the third-party library, and the first tensor information and the second tensor information are converted using a distributed tensor memory representation data structure as the conversion medium; a control module, used by the third-party library operator to send computing instructions to multiple devices respectively according to the second tensor information, the computing instructions including the third-party distributed computing task, so that the multiple devices use their stored first tensor data to execute the third-party distributed computing task, and the multiple devices output the second tensor data after executing the third-party distributed computing task, the second tensor data being stored in a second data structure.

[0016] In one possible implementation, the determining module is further configured to: in response to the output of second tensor data after multiple devices perform a third-party distributed computing task, obtain third tensor information, the third tensor information including: metadata of the second tensor data and second distribution information, the second distribution information representing the distribution information of the second tensor data distributed across multiple devices in a second data structure; and based on the third tensor information, convert to obtain fourth tensor information, the fourth tensor information represented by a distributed tensor memory representation data structure, the fourth tensor information being used by the deep learning framework to continue distributed computing.

[0017] In one possible implementation, after the control module converts the third tensor information to obtain the fourth tensor information, it is further configured to: send a data rearrangement request to multiple devices, the data rearrangement request including the fourth tensor information and graph inference arrangement information, so that the devices rearrange the second tensor data after receiving the data rearrangement request, so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

[0018] In one possible implementation, the first tensor information also includes the stream information corresponding to the device.

[0019] In one possible implementation, the device further includes: an operation module; the operation module is used to acquire user operation information and set stream information based on the user operation information.

[0020] Thirdly, this application provides a distributed system including at least one host and multiple devices, each host or device including a processor and a memory; the processor of at least one host is used to execute instructions stored in the memory of at least one host to cause at least one host to perform the method provided in the first aspect; the processor of at least one device is used to execute instructions stored in the memory of at least one device to cause the multiple devices to perform: receiving computation instructions sent by the host, and performing a distributed computation task of a third-party library operator using first tensor data stored therein according to the computation instructions.

[0021] Fourthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method provided in the first aspect.

[0022] Fifthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, cause the computing device to perform the method provided in the first aspect; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method provided in the first aspect. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0023] Sixthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method provided in the first aspect, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method provided in the first aspect. For example, a cluster of computing devices may include one or more computing devices.

[0024] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the execution process of an AI framework calling a third-party library in a related technology provided in this application embodiment;

[0026] Figure 2 This is a comparison diagram of NdArray and DLPack data type conversion schemes in a single-card scenario provided by an embodiment of this application;

[0027] Figure 3This is a schematic diagram illustrating an application scenario where an AI framework accesses a third-party library for distributed computing, as provided in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the architecture of a distributed system provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a distributed tensor memory representation data structure, ShardTensor, provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a DLPack provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram illustrating a scenario of a third-party library invocation method based on a distributed system, as provided in an embodiment of this application.

[0032] Figure 8 This is a comparison diagram between a solution provided in this application and a traditional solution;

[0033] Figure 9 This is a comparison diagram of a stream in a traditional solution provided by an embodiment of this application;

[0034] Figure 10 This is a logical diagram of a compilation-stage reasoning provided in an embodiment of this application;

[0035] Figure 11 This is a schematic diagram of the execution flow of inserted data rearrangement provided in an embodiment of this application;

[0036] Figure 12 This is a schematic diagram of a MindSpore calling a third-party library according to an embodiment of this application;

[0037] Figure 13 This is a schematic diagram of the interaction between a host and a device provided in an embodiment of this application;

[0038] Figure 14 This is a schematic diagram of the structure of a third-party library calling device based on a distributed system, provided in an embodiment of this application;

[0039] Figure 15 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0040] Figure 16 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0041] Figure 17This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0042] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0043] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0044] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0046] To facilitate understanding of the solutions provided in the embodiments of this application, the relevant technical terms involved in the embodiments of this application will be introduced first.

[0047] AI: refers to the intelligence exhibited by machines created by humans. In this application, AI mainly refers to the intelligence exhibited by models implemented through deep learning.

[0048] Host: refers to the core component of a computing system that is responsible for controlling and scheduling tasks. It usually runs the main program and sends computing tasks to devices for execution. The host is usually the CPU (Central Processing Unit).

[0049] Device: refers to the hardware in a computing system responsible for performing computing tasks. It is generally a parallel computing card such as GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), and NPU (Neural Processing Unit). It waits for the host to send code and executes it.

[0050] Distributed system: refers to a system composed of multiple nodes that collaborate to complete tasks through network connections, the method of which is not limited. In this application, nodes refer to host and device, and a distributed system consists of at least one host and multiple devices.

[0051] Distributed Parallel: This refers to a computation process that is executed simultaneously on multiple devices. These devices synchronize and communicate with each other as needed to complete the entire computation process. In this case, the input data, intermediate data, and output data of the computation process may all be distributed across multiple devices.

[0052] AI frameworks (deep learning frameworks) are software tools or platforms used for designing, training, and validating AI algorithm models. They provide the infrastructure for building, training, and deploying AI models. They abstract away the underlying hardware and complex implementation details, allowing developers to focus on algorithm design and model development without directly dealing with the low-level details. Common AI frameworks include TensorFlow, PyTorch, MindSpore, and JAX. Different AI frameworks may use different data structures in their implementations. These data structures are designed primarily to optimize computational performance, ensure hardware compatibility, and improve the development experience.

[0053] Third-party libraries refer to software libraries created by third-party developers. These libraries provide program modules with different functionalities for processing data or performing specific mathematical and logical operations in computing tasks. They are available for developers to integrate into their own applications or for other developers to use in their projects, thereby reducing development time, improving efficiency, and enabling complex functionalities. The third-party libraries used in this application are distributed computing third-party libraries, meaning third-party libraries that can operate in parallel across multiple devices, or simply third-party distributed libraries. These are independent software libraries capable of coordinating computing tasks and accelerating model training or inference on multiple devices (such as multiple GPUs). These libraries are developed by third parties (not native to the AI ​​framework) and focus on specific functions such as high-performance computing, distributed architecture, and optimization algorithms. When a third-party library is called and integrated into the AI ​​framework, it is encapsulated as an operator on the host machine; in this application embodiment, this is referred to as a third-party library operator.

[0054] Tensor: Essentially a high-dimensional array. In distributed computing scenarios, a large tensor is typically divided into multiple blocks and distributed across multiple devices for computation and storage. Each device stores a portion of the tensor's data; this portion stored on each device can be called a local tensor. Most AI frameworks use tensors as their basic data structure, but different AI frameworks may use different types of tensor data, and different types of tensor data structures differ in their underlying storage and design.

[0055] A stream is an execution queue used to organize and schedule the execution order of operations on devices such as GPUs. It defines the flow of operations on the device, ensuring that operations within the stream are executed sequentially. Tasks within the same stream are executed serially, while asynchronous tasks on different streams can be executed in parallel.

[0056] Data layout refers to how tensors are stored in memory. Common data layouts include NCHW (batch, channel, height, width) and NHWC (batch, height, width, channel). Different frameworks and libraries may have different default data layouts. For example, TensorFlow often uses NHWC layout, while PyTorch usually uses NCHW layout.

[0057] Graph reasoning is a reasoning method based on a computation graph, commonly used in AI frameworks. It refers to the process of executing operators on nodes sequentially through a pre-trained computation graph to ultimately generate the model's output. For example, in the inference phase, given input data, the model derives output data based on the computation graph.

[0058] The following explains how AI frameworks call third-party library operators in related technologies.

[0059] like Figure 1 As shown, the solution for integrating AI frameworks with third-party libraries mainly uses a common intermediate data structure to convert data types between different frameworks, thus solving the problem of data type mismatch between the data structures used by AI frameworks and third-party libraries.

[0060] Figure 1A illustrates a single-card scenario, i.e., one host and one device. In this scenario, the computation of operators is performed by a single device. The AI ​​framework MindSpore uses a custom interface to access the third-party library JAX operators. MindSpore performs computations based on Tensor data structures, while JAX performs computations based on JaxArray data structures. That is, the input and output data of MindSpore operators are both Tensor data structures, while the input and output data of JAX operators are both JaxArray data structures. These two data structures are incompatible. Before calling the JAX library, it is necessary to convert the Tensor data structure output by MindSpore's previous AI framework operators to an NdArray data structure (because most frameworks support conversion to and from NdArray). The output data of the MindSpore Tensor data structure is converted into a JaxArray data structure. Therefore, the Custom interface will convert the output data of the MindSpore Tensor data structure into an NumPy NdArray before calling the third-party library. After calling the third-party library, the execution result of the third-party library operator is first converted into the NumPy NdArray format, and then converted back into the MindSpore Tensor data structure. The output data of the Tensor data structure can then be used to execute subsequent AI framework operators. All these data conversion processes need to be performed on the host side. When the output data of the operator is located on the device side, the data on the device side needs to be moved to the host side for data conversion, and then moved back to the device side to execute the next operator calculation.

[0061] Figure 1B illustrates the execution process of an AI framework calling a third-party library in a distributed scenario, specifically a scenario with at least one host and multiple devices. This distributed scenario involves distributed computation performed by four devices (device0, device1, device2, and device3). Here, f() represents the AI ​​framework's operator, and C() represents the third-party library's operator. Different colored boxes within the operator represent tensor data distributed across different devices. Since the third-party library and the AI ​​framework use different tensor data structures, and the data conversion process can only be performed on the host side, multiple data transfers are required to convert the AI ​​framework's data type to the third-party library's data type: ① After the AI ​​framework's operator computation is completed, the tensor data must first be transferred from the device side to the host side (i.e., D2H) and then converted on the host side into the intermediate data structure NumPy. The data is first transferred from the host side to the device side (H2D) to form an array, then converted to a third-party library data type. This tensor data of the third-party library data type is then moved from the host side to the device side for distributed parallel computation across multiple devices. The resulting tensor data of the third-party library data structure output by the operator needs to be moved back to the host (D2H) for data type conversion. First, it is converted to an intermediate data structure, NumPy Array, then to the AI ​​framework data type, and finally moved back to the device side (H2D) for the AI ​​framework to execute the next operator's computation. In short, to make the AI ​​framework's output data recognizable by the third-party library, the data must first be moved from the device side to the host side, converted using the common intermediate data structure NumPy Array, and then moved back to the device side for computation. Similarly, the computation results of the third-party library also need to undergo data movement and type conversion before they can be recognized by the AI ​​framework. The entire computation process involves four data movements. This repeated movement between the host and device sides wastes computational resources and affects computation speed.

[0062] In another related technology, to avoid the repeated data transfer between the host and device sides during data type conversion, a method of using a memory-represented data structure as an intermediate data structure has been proposed, such as DLPack. DLPack is a standardized memory-represented data structure that allows two different AI frameworks (third-party libraries can be regarded as AI frameworks) to exchange tensor data address information. This data conversion process is essentially a conversion of the tensor data representation form, without involving the conversion of the data structure stored on the device itself. Therefore, no data transfer is required during the conversion process.

[0063] Figure 2This diagram compares the data type conversion schemes of NdArray and DLPack in a single-card scenario. The tensor consists of two parts: metadata ( Figure 2 The tensor consists of a meta field and storage (the address where the data is stored). Based on NdArray type conversion, when converting the data structure type, the storage structure of the tensor data needs to be changed. This conversion needs to be performed on the host side. Therefore, the tensor data 11 of the Tensor data structure needs to be copied from address 1 on the device to address 2 on the host side, converted to tensor data 12 of the NdArray data structure, then converted to tensor data 13 of the JAXArray data structure from the third-party library, and finally copied back from the host side to address 3 on the device. Based on DLPack type conversion, it is essentially a conversion of the tensor's memory representation. When converting tensor data 21 of the Tensor data structure to tensor data 22 of the DLPack data structure, only the address value of the storage is copied. Similarly, when converting tensor data 22 of the DLPack data structure to tensor data 23 of the JAXArray data structure, only the address value of the storage is copied. The tensor data is always stored at address 1 on the device, without involving data movement.

[0064] However, DLPack currently only considers single-card scenarios, where all the data of a tensor is stored on a single device. In distributed scenarios, tensors are distributed across multiple different devices, and DLPack cannot describe the distribution information of tensors across different devices. This makes it impossible for AI frameworks to call third-party libraries in distributed scenarios.

[0065] This application provides a method for calling a third-party library based on a distributed system. This method uses a distributed tensor memory representation data structure as an intermediate data structure in the data type conversion process. It can transfer the distribution information and metadata of tensors across multiple devices between different frameworks, avoiding the need to move tensor data between the host and devices, thus preventing wasted computational performance and quickly enabling third-party distributed library operators to be integrated into the AI ​​framework's computation. In this method, after the deep learning framework executes a previous operator, the host in the distributed system can collect the first tensor information of the first tensor data from the output data of that operator. This first tensor information includes the first distribution information of the first tensor data distributed across multiple devices and the metadata of the first tensor data. The host uses the distributed tensor memory representation data structure as an intermediate data structure to convert the first tensor information into second tensor information supported by the third-party library. This allows the first distribution information and metadata to be transferred between the deep learning framework and the third-party library, enabling the third-party library to continue executing its computational tasks based on the first distribution information of each device and the first tensor data stored on that device. This realizes the calling of third-party libraries by AI frameworks in a distributed system scenario.

[0066] The technical solution provided in this application is described below.

[0067] First, the application scenarios of the embodiments of this application will be described.

[0068] For example, Figure 3This illustration shows an application scenario where an AI framework integrates a third-party library for distributed computing. In this scenario, after the AI ​​framework's native operator 1 performs distributed computing on devices 0, 1, 2, and 3, the tensor data output by native operator 1 is distributed and stored across these devices. Since the AI ​​framework performs computation based on a first data structure, the tensor data output by the native operator is stored using this first data structure. To easily distinguish the tensor data output by operators from different frameworks, this embodiment refers to the tensor data required by the third-party distributed computing task within the tensor data output by the AI ​​framework's native operator 1 as the first tensor data. For algorithm implementation purposes, the next computation task needs to call a third-party library. After being integrated into the AI ​​framework, it will be encapsulated into an operator, namely a third-party library operator. This operator can be used for distributed computing. The third-party library operator continues to use the first tensor data to perform distributed computing on device0, device1, device2, and device3. After the computing is completed, the output tensor data is distributed and stored in device0, device1, device2, and device3. Here, the tensor data output by the third-party library operator is called the second tensor data. Since the third-party library performs computing based on the second data structure, the second tensor data is stored based on the second data structure. The second tensor data will continue to be used for distributed computing of the AI ​​framework's native operator 2, and the third-party library call ends.

[0069] The following is combined with Figure 3 The application scenarios shown illustrate the distributed system of this application embodiment.

[0070] For example, Figure 4 A schematic diagram of the architecture of a distributed system 100 provided in an embodiment of this application is shown. It should be understood that... Figure 4 The distributed system 100 shown is merely illustrative and does not constitute a limitation of this application.

[0071] like Figure 4 As shown, the distributed system 100 consists of host0, host1, device0, device1, device2, and device3. This is understandable. Figure 4Although the distributed system 100 shown includes 2 hosts and 4 devices, the embodiments of this application are not limited to this, and may include other numbers of hosts and devices. In the embodiments of this application, the hosts and devices of the distributed system 100 can communicate through a network, which serves as a medium to provide communication links between hosts, between hosts and devices, and between devices. The network can include various connection types, including direct connections and indirect connections.

[0072] In one possible implementation, each host and each device can be directly connected to form a communication connection. For example, host0 establishes communication connections with device0, device1, device2, and device3. Alternatively, each host can be directly connected to some devices to form a communication connection. For instance, host0 can be directly connected to device0 and device2, and host1 can be directly connected to device1 and device3. Hosts can also be directly connected to other hosts, and devices can be directly connected to other devices. In other embodiments, the above-mentioned communication connections can also be formed through indirect connections. The connection relationships between hosts and devices described above are only illustrative examples. In practical applications, the connection relationships between hosts and devices in a distributed system can be adjusted according to the actual situation. Multiple hosts and devices can reside on the same physical machine or on different physical machines. The distributed system can be fully deployed in the cloud or partially deployed in the cloud.

[0073] After the native operator 1 of the AI ​​framework performs distributed computing and outputs the first tensor data, such as Figure 4 As shown, the first tensor data T is divided into multiple tensor blocks [T[0], T[1], T[2], T[3]], which are distributed and stored on device0, device1, device2, and device3. The distribution information of each tensor block on the device is called the first distribution information. The distribution information may include the storage address of each tensor block on the device, the topology between each tensor block, the size of the memory occupied by each tensor block, the identifier of the tensor block, and the relevant information of the device where the tensor block is stored, etc. The tensor blocks are stored in the first data structure. The meta-information of the tensor data is the data that describes the tensor attributes, which may include information such as the shape, dimension, and data type of the tensor.

[0074] Since third-party library operators cannot recognize the first data structure and cannot obtain the distribution information and meta-information of the first tensor data, they cannot use the first tensor data on multiple devices to perform distributed computing. In this embodiment, a distributed tensor memory representation data structure is used as a conversion medium between the AI ​​framework and the data structure of the third-party library. This can transfer the distribution information of the tensor data stored on multiple devices between different frameworks, so that third-party library operators can use the first tensor data to perform distributed computing.

[0075] The distributed tensor memory representation data structure provided in the embodiments of this application is described below.

[0076] The distributed tensor memory representation data structure provided in this embodiment is a standardized cross-framework data structure for representing tensors. Each framework can use this data structure to transform its internal tensor representation without changing the storage method of the tensor data. The distributed tensor memory representation data structure can be converted to and from a first data structure and a second data structure, respectively. The first and second data structures are used to distinguish between different data structures used by AI frameworks and third-party libraries. It is understood that the distributed tensor memory representation data structure can also be converted to other types of tensor data structures. This conversion process can be viewed as a re-encapsulation or re-description of tensor information, without the need to reallocate tensor storage memory on the device or copy data.

[0077] The distributed tensor memory representation data structure contains the metadata of the tensor and the distribution information of the tensor across multiple devices, thus enabling the transmission of the distributed information of the tensor to other frameworks through a transformation process. This application does not limit the specific implementation form of the distributed tensor memory representation data structure.

[0078] For example, Figure 5 This illustration shows a schematic diagram of a distributed tensor memory representation data structure, ShardTensor, provided by an embodiment of this application. Figure 5 As shown, ShardTensor mainly consists of MetaTensor and Devicesarrangement.

[0079] MetaTensor is used to store the meta-information of tensors, specifically including: Shape: the shape of the tensor; Dtype: detailed information about the tensor's data type; Ndims: the number of dimensions of the tensor.

[0080] Specifically, Dtype includes: Type: data type, such as integer (Int), floating-point (Float), etc.; Bits: number of bits of the data type; and Lines: number of channels of the data type.

[0081] DeviceSarrangement is used to store the distribution information of tensors across multiple devices. Specifically, it includes: mesh: information for each device, including a list of DeviceInfo; ReplicateDims: used to indicate the partitioning method of the tensor.

[0082] Each DeviceInfo in the DeviceInfo list describes specific device information, including: device_address: the tensor storage address on the device; device_streams: the stream information of the device; device_type: the type of the device (such as CPU, NPU, GPU, etc.); device_id: the specific number of the device.

[0083] In other possible implementations, the distributed tensor memory representation data structure can also be improved by adding an array structure to the tensor memory representation data structure. This array structure is used to represent the distribution information of different blocks of tensor data on different devices, thus improving it into a distributed tensor memory representation data structure. Figure 6 Taking the tensor in-memory representation data structure DLPack as an example, the core data structure DLTensor of DLPack contains: the tensor's metadata (such as Shape and Dtype) and DLDevice. DLDevice contains basic information about the tensor and a single device, including device_type and device_id. An array structure can be added to DLPack to store the distribution information of tensors across multiple devices, resulting in a distributed tensor in-memory representation data structure. This allows different frameworks (such as PyTorch, TensorFlow, MXNet, JAX, MindSpore, etc.) to efficiently share tensor data in distributed scenarios without additional copy operations.

[0084] Next, the method for calling third-party libraries based on distributed systems provided in the embodiments of this application will be described in detail.

[0085] For ease of description, the following will use the host as the execution subject. For example... Figure 7 The scenario diagram shown illustrates that the method includes the following processing.

[0086] First, identify the third-party distributed computing task.

[0087] In this step, after the previous AI box operator performs distributed computation, a third-party distributed computation task can be determined based on the static or dynamic computation graph. This third-party distributed computation task is executed by a third-party library operator, and its computation object is the first tensor data. This first tensor data can be the tensor data required by the third-party distributed computation task from the tensor data output after the previous AI box operator's distributed computation.

[0088] This step can also obtain the first tensor information corresponding to the first tensor data. The first tensor information includes the metadata and first distribution information of the first tensor data. The first distribution information represents the distribution information of the first tensor data distributed across multiple devices using a first data structure supported by the deep learning framework. After the AI ​​framework operator outputs the first tensor data on the device, the device will return the first tensor information related to the first tensor data stored on it to the host controlling the device. Obtaining the first tensor information can be performed after the AI ​​framework operator outputs the first tensor data after performing distributed computation.

[0089] Figure 7 In this embodiment, the host obtains the first tensor information from the devices it controls. This embodiment does not limit the method of obtaining the first tensor information; in other embodiments, the host can obtain the first tensor information from other hosts and devices. For example, each host can obtain the first tensor information from the devices it controls, and also from other hosts the first tensor information of devices controlled by other hosts. This embodiment does not limit the method by which a host obtains the first tensor information from other hosts. For instance, when each host stores the first tensor information of the devices it controls, the allGather operation can be used to collect the first tensor information. AllGather is a collective communication operation in which each host sends its own first tensor information to other hosts, and ultimately each host receives the first tensor information sent by all other hosts.

[0090] For example, such as Figure 4In the distributed system 100 shown, with host0 controlling device0 and device1, and host1 controlling device2 and device3, host0 stores the first tensor information of device0 and device1, and host1 stores the first tensor information of device2 and device3. After the allGather operation, both host0 and host1 store the first tensor information of device0, device1, device2 and device3 respectively.

[0091] Then, based on the first tensor information, the second tensor information is obtained.

[0092] The second tensor information is represented by a second data structure supported by a third-party library. The second tensor information includes the metadata of the first tensor data and the first distribution information of the first tensor data across multiple devices.

[0093] In this step, the first tensor information can be converted into intermediate tensor information represented by a distributed tensor memory representation data structure. For example, an interface can be called to convert the first tensor information into an intermediate tensor information represented by a distributed tensor memory representation data structure. Figure 5 The intermediate tensor information represented by the ShardTensor shown can be transformed into the second tensor information again because the distributed tensor memory representation data structure can be repackaged into other data structures. Thus, the distributed tensor memory representation data structure is used as a conversion medium between the first and second data structures to realize the transfer of tensor information.

[0094] Then, the second tensor information is input into the invoked third-party library operator, and the third-party library operator running on the host sends calculation instructions to the multiple devices respectively based on the second tensor information.

[0095] The computation instructions include the third-party distributed computing task, which controls multiple devices to execute the third-party distributed computing task using the first tensor data they store. After executing the third-party distributed computing task, the multiple devices output second tensor data, which is stored in the second data structure.

[0096] The second tensor information is represented using a second data structure supported by the third-party library, which can be used to identify the third-party library and perform distributed computation. When the third-party library is called by the AI ​​framework, it is encapsulated as a third-party library operator. The third-party library operator resides in the host and can be pre-stored in the host's memory. When the third-party library operator is called, it is loaded into memory and executed on the processor. The host inputs the second tensor information into the third-party library operator, calls the third-party library operator's interface, and the third-party library operator running on the host sends computation instructions to multiple devices. These computation instructions can initiate distributed computation tasks of the third-party library operator on the devices.

[0097] In one example, the host may launch a computation graph containing the AI ​​framework operators and third-party library operators to the device before the AI ​​framework operators perform computations. In this step, the computation instructions contain code that starts the distributed computation task of the third-party library operators on the device.

[0098] In another example, the execution code for a distributed computing task of a third-party library operator can be launched to the device via computation instructions, enabling the device to execute the distributed computing task. For instance, the computation instructions could include the address of a function in the distributed computing task to be executed by the device, the device ID, and the address of the first tensor data required by the function.

[0099] In this embodiment, when the host calls a third-party library operator, it may load the user-written third-party library operator from memory and run the third-party library operator in the processor. In other embodiments, it may request a remote server, receive the third-party library operator sent by the remote server, and run the third-party library operator in the processor.

[0100] In some embodiments, when a third-party library operator running on a host sends computation instructions to multiple devices, different hosts may send corresponding computation instructions to devices controlled by that host. For example, using Figure 4Taking the distributed system shown as an example, with host0 controlling device0 and device1, and host1 controlling device2 and device3, host0 sends computation instructions to device0 and device1, and host1 sends computation instructions to device2 and device3. For example, the computation instruction sent by host0 to device0 contains the first distribution information of T[0] stored in device0, the computation instruction sent by host0 to device1 contains the first distribution information of T[1] stored in device1, the computation instruction sent by host1 to device2 contains the first distribution information of T[2] stored in device2, and the computation instruction sent by host1 to device3 contains the first distribution information of T[3] stored in device3, thereby controlling multiple devices to perform distributed computation using their stored first tensor data according to the corresponding distribution information.

[0101] In this embodiment, after the AI ​​framework's operator performs distributed computation and outputs the first tensor data, it performs data structure transformation through a distributed tensor memory representation data structure so that the first tensor data can be recognized by the third-party library operator. Similarly, after the third-party library operator performs distributed computation, the second tensor data output by the third-party library operator and distributed storage on multiple devices also needs to be transformed so that it can be recognized by the next operator of the AI ​​framework.

[0102] In some embodiments, such as Figure 3 As shown, after inputting the second tensor information into the invoked third-party library operator, the second tensor data output by the third-party library operator can continue to be used for distributed computing of the AI ​​framework's native operator 2. Following the above embodiment, the following processing can be performed: multiple devices execute the third-party distributed computing task and output the second tensor data to obtain the third tensor information. The third tensor information includes the second distribution information and the metadata of the second tensor data, which are distributed across multiple devices. The second tensor data is stored in a second data structure. Based on the third tensor information, a fourth tensor information is obtained, represented by a distributed tensor memory representation data structure. This fourth tensor information is used by the AI ​​framework to continue distributed computing.

[0103] In this embodiment, the host can collect the second distribution information and metadata of the second tensor data stored on different devices to obtain the third tensor information. For example, the host can obtain the third tensor information from other hosts and their related devices. The specific acquisition method can refer to the process of obtaining the first tensor information described above, and will not be repeated here. The second tensor data is stored in a second data structure, and the third tensor information is also represented in a second data structure. Then, the third tensor information is integrated and transformed to obtain the fourth tensor information. The fourth tensor information contains the metadata of the second tensor data and the distribution information of the second tensor data on multiple devices. The fourth tensor information is represented using a distributed tensor memory representation data structure, which can be converted into a first data structure that the AI ​​framework can recognize, so that the second tensor data can continue to be used for the next operator of the AI ​​framework to continue distributed computing.

[0104] Figure 8 The diagram illustrates a comparison between the third-party library invocation scheme based on the distributed system in this embodiment and the traditional scheme. Framework 1 refers to the native AI framework used by the host, and Framework 2 refers to the third-party library. The commonality between the two is that, before the first computation of the operator in Framework 1, the initial tensor data to be computed is stored on the host side and needs to be loaded onto the device side—this is the first data transfer in Framework 1. Then, the operator in Framework 1 performs computation based on the initial tensor data and outputs the first tensor data.

[0105] Different frameworks use different data types to represent tensors. Traditionally, NumPy's NDArray is used as a bridge to convert between the two frameworks' data types. However, since tensor computations need to be performed on the device while data type conversions need to be performed on the host, this approach requires frequent data transfer. Specifically, after the computation is performed in framework 1, the first tensor data on the device needs to be transferred to the host and converted into an NdArray data structure. When input to framework 2, it is then converted into the data structure used by framework 2 and transferred back to the device to perform the computation of framework 2's operators. Similarly, after framework 2 performs the computation and outputs the second tensor data, the second tensor data on the device needs to be transferred to the host and converted into an NdArray data structure. When input to framework 1, it is then converted into the data structure used by framework 1 and transferred back to the device to perform the computation of framework 1's operators.

[0106] This solution uses a distributed tensor memory representation structure to transfer the distribution information of tensors across multiple devices between different frameworks without changing the storage format of tensor data on the devices. The distributed tensor memory representation structure is used as an intermediate data structure to complete the conversion between the two data types. After the calculation is performed in framework 1, the calculation in framework 2 can continue without data transfer, effectively reducing the time spent on data transfer and the computational performance required.

[0107] In some embodiments, the first tensor information may also include the stream information corresponding to the device, so that the computation instructions sent by the host to the device may also include the stream information corresponding to the device.

[0108] Among them, Stream information refers to the information of the streams related to the task execution of the device created by the AI ​​framework. A stream is an execution queue, and the stream corresponding to each device defines the execution flow on that device. The device will execute the operations in the stream in sequence.

[0109] Each framework creates and maintains its own stream to ensure that operator instructions are correctly executed by the device. For example... Figure 9 As shown, in the traditional approach, Framework 1 creates and maintains its own stream. When executing the computation of Framework 1's operator, the device executes the operations in that stream sequentially. When calling the operator of Framework 2, Framework 2 (i.e., the third-party library) creates and maintains its own new stream. When executing the computation of Framework 2's operator, the device executes the operations in this newly created stream sequentially. Since some memory resources are allocated to the new stream when it is created, the new stream needs to be destroyed after the operator of Framework 2 performs the computation to release the resources occupied by the stream and prevent the system memory from being gradually exhausted. Thus, there is a certain overhead of creating and destroying streams each time the third-party library is called. This overhead is generally in the millisecond range, but it can have a significant impact on computational performance when the third-party library is called frequently.

[0110] In this embodiment, a distributed tensor memory representation structure is used to pass the stream of each device in framework 1 to framework 2. During the calling process, framework 2 can continue to use the stream created and maintained by framework 1, which can avoid the creation and destruction of new streams in framework 2 and save overall execution time.

[0111] For example, when obtaining the first tensor information, the device's stream information and first distribution information can be obtained from other hosts. When integrating the first tensor information into a distributed tensor memory representation structure, an array structure for managing the stream of each device can be added. For example, such as... Figure 5 The `device_streams` array in the `ShardTensor`'s `DeviceInfo` stores the stream information corresponding to each device. This allows the two frameworks to synchronize distributed tensors and exchange streams when converting data types using the distributed tensor memory representation structure. When sending computation instructions to multiple devices, the host sends the instructions to each device, which also include the stream information corresponding to that device. This controls each device to perform distributed computation using its stored first tensor data based on the stream information. In this way, the two frameworks can share stream information, and third-party library operators can use the streams created and maintained by the AI ​​framework when performing computations.

[0112] In some embodiments, user operation information can also be obtained, and based on the user operation information, the first tensor information, the second tensor information, or the stream information in the distributed memory data representation structure can be set.

[0113] This embodiment does not restrict the method of obtaining user operation information. For example, user operation information can be received through the API (Application Programming Interface) provided by the AI ​​framework. Users can directly manipulate the stream information in the distributed tensor memory representation data structure through the API. Alternatively, user operation information can be obtained by parsing the configuration file provided by the user, without the need to manually input the operation information.

[0114] User operation information can be used to set stream information, such as the task execution order or priority, task dependencies, synchronous and asynchronous mechanisms, and task scheduling strategies. This not only helps users manage collaborative operations between multiple devices, but also manages synchronous and / or asynchronous operations between different frameworks. Since the two frameworks share stream information, the task queue in the stream on each device can contain tasks from AI framework operators and third-party library operators. This allows the stream on each device to independently schedule local tasks and also achieve cross-device and cross-framework stream collaboration, improving the efficiency and flexibility of distributed computing.

[0115] The compilation phase of an AI framework typically refers to the process of compiling the AI ​​framework's operators and computation graph into code that can be efficiently executed on the device. During this process, the compiler needs to know in advance the shape and data type of the operator input and output data, as well as the data arrangement (such as block or row partitioning) to ensure correct compilation. It's worth noting that in scenarios involving third-party libraries, if the AI ​​framework has a compilation phase, it also needs to consider the output shape and data type (dtype) of the third-party library's operators, as well as the data arrangement of the third-party library's input and output data. This ensures that the AI ​​framework knows the output characteristics of the third-party library's operators in advance, facilitating optimization of the computation graph during compilation and avoiding data errors and unnecessary performance overhead.

[0116] Generally, the logic of reasoning during the compilation phase can be as follows: Figure 10 As shown, the inference logic for Shape and Dtype can be provided by the user or determined by provided rules, while the inference logic for data arrangement is determined by the data arrangement of the context (i.e., the preceding and following operators) of the third-party library operator. However, in a distributed scenario where a third-party library is called, due to inconsistencies between the data sharding and memory layout of the third-party library operator output data and the AI ​​framework, the data arrangement of the second tensor data output after the third-party library calculation may not actually be determined based on the context. To avoid this affecting subsequent calculations, this embodiment can insert a data rearrangement after converting the third tensor information to obtain the fourth tensor information, to ensure that the result output by the third-party library operator is consistent with the result of the AI ​​framework graph inference.

[0117] In this embodiment, the host sends data rearrangement requests to multiple devices. These requests include fourth tensor information and graph inference arrangement information. Upon receiving the data rearrangement request, the devices rearrange the second tensor data so that the rearranged data arrangement information matches the graph inference arrangement information. The graph inference arrangement information refers to the input data arrangement information of the next operator obtained through graph inference in the AI ​​framework's computation graph. By inserting a rearrangement operation, the data arrangement format of the second tensor data can be adjusted to match the format desired by the AI ​​framework's computation graph, thereby ensuring the correctness of the entire inference process.

[0118] Figure 11The diagram illustrates an execution flow for inserting data rearrangement. After the computation task of the previous operator (i.e., the native operator 1 of the AI ​​framework) is completed, the output Tensor (i.e., the first tensor data) of the operator is distributed across multiple devices. The host collects the distribution information such as the storage address of the Tensor on the device and stream information through the allgather operation, and integrates them into a complete distributed tensor memory representation structure ShardTensor. Then, the ShardTensor is converted into a third-party library distributed Tensor supported by a third-party library. At this point, the third-party library distributed Tensor can be input into the called third-party library operator on the host to control the device to perform distributed computation. The third-party library distributed Tensor of the computation output second tensor data is then encapsulated into a ShardTensor, where data rearrangement is inserted to obtain the rearranged out_shard. The data arrangement of out_shard is consistent with the input format expected by the next operator, so that the subsequent operator can correctly process the second tensor data.

[0119] In other embodiments, if the data rearrangement is only a logical transformation and does not involve actual data rearrangement, such as transposition, block reorganization, and data format conversion, the data rearrangement may only change the data arrangement information in the distributed tensor memory representation structure ShardTensor. In this case, it is not necessary to send data rearrangement requests to multiple devices, but to change the data arrangement information in ShardTensor on the host side. For example, a new layout can be simulated by changing the offset and dimension information of the memory representation, without moving the actual data.

[0120] In some embodiments, after the second tensor information is input into the invoked third-party library operator, the third-party library may also rearrange the data according to the data arrangement requirements of its operator for the input data, so that the data arrangement of the first tensor data is consistent with the input format expected by the third-party library operator, so that the third-party library operator can correctly process the first tensor data.

[0121] Next, we will use specific examples to illustrate the third-party library invocation method based on a distributed system provided in the embodiments of this application.

[0122] Taking the MindSpore AI framework's call to third-party libraries in a distributed scenario as an example, MindSpore provides Custom operators to support third-party libraries. Custom operators are the unified interface for MindSpore to call third-party library operators. This embodiment improves upon MindSpore's existing Custom operators by defining the MSLibAdapter interface. This interface is used to convert between distributed tensor memory representation data structures and data structures supported by third-party libraries, enabling support for distributed and parallel third-party libraries. Figure 12 The flowchart shown illustrates that the method includes the following steps:

[0123] Step 121: Identify the third-party distributed computing task.

[0124] After the previous MindSpore operator is executed, a third-party distributed computing task can be determined based on the computation graph. If the next step is a third-party distributed computing task, the following steps continue. If the next step is still a distributed computing task of MindSpore's own operator, the following steps are not required. After MindSpore executes the previous operator in a distributed scenario, it outputs the first tensor data of the first data structure. In this step, the first tensor information is also obtained. The host uses the allGather operation to collect the first tensor information output by the previous operator. The first tensor information mainly includes the address of the first tensor data on the device, topology, memory size, device-related information, and other distribution information, as well as stream information.

[0125] Step 122: Based on the first tensor information, integrate to obtain the intermediate tensor information.

[0126] After obtaining enough information from the first tensor, the Host can construct the second tensor information of the ShardTensor structure.

[0127] Step 123: Based on the intermediate tensor information, convert it to obtain the second tensor information.

[0128] After the construction is complete, the host calls the MSLibAdapter interface to encapsulate the intermediate tensor information of the ShardTensor structure and transform it into the second tensor information of the second data structure supported by the third-party library.

[0129] Step 124: Input the second tensor information into the called third-party library operator.

[0130] The host inputs the second tensor information into the user-written third-party library operator. At this point, the third-party library can rearrange the first tensor data according to the data arrangement requirements of its operator for the input data.

[0131] In step 125, the third-party library operator running on the host sends computation instructions to multiple devices based on the second tensor information.

[0132] Based on the device indicated by the second tensor information, the third-party library operator running on the host sends computation instructions to multiple devices, so that the multiple devices use the first tensor data to execute the distributed computation task of the third-party library operator using the stream used by the previous operator.

[0133] Step 126: Obtain the third tensor information.

[0134] After the calculation is completed, the third-party library operator outputs the second tensor data of the second data structure, and the host collects and organizes the third tensor information. The third tensor information mainly includes the distribution information of the second tensor data on the device, such as the address, topology, memory size, and device-related information, as well as stream information and the metadata of the second tensor data.

[0135] Step 127: Based on the third tensor information, convert to obtain the fourth tensor information.

[0136] The Host calls the MSLibAdapter interface again, and the third tensor information is converted back into the fourth tensor information of the ShardTensor structure.

[0137] Step 128: Send a data rearrangement request to the device.

[0138] The second tensor data is rearranged to meet the requirements of the next operator in the Mindspore framework for input data arrangement before proceeding to the next operator's calculation.

[0139] In a distributed scenario, the distributed memory data representation structure provided in this embodiment can greatly reduce the data transfer overhead between frameworks, and can also pass the stream of each device between different frameworks, thereby avoiding the creation and destruction of new streams in third-party library operators and saving overall execution time.

[0140] Based on the above embodiments, the following example, using a distributed system 100 as an example, illustrates the flow of the third-party library invocation method provided in this application embodiment. To facilitate the description of the interaction process between the host and device, in this example, host0 in the distributed system 100 is set to control device0, device1, device2, and device3. It is understood that this control relationship does not constitute a restriction on the control relationship between the host and device in the distributed system. Figure 13 As shown, the method includes the following steps.

[0141] Step 131: The host identifies the third-party distributed computing task and obtains the first tensor information.

[0142] Among them, device0, device1, device2 and device3 can send the first distribution information of the first tensor data stored locally to the host that controls the device, so that the host can perform an allgather operation to finally obtain the complete first distribution information.

[0143] Step 132: The host converts the first tensor information to obtain the second tensor information.

[0144] For example, the host collects the complete first tensor information, integrates it to obtain intermediate tensor information, and then transforms it to obtain the second tensor information.

[0145] In other embodiments, when there are multiple hosts in the distributed system, each host may collect the complete first tensor information, integrate the first tensor information to obtain intermediate tensor information, and then encapsulate the intermediate tensor information to obtain the second tensor information. Alternatively, fewer hosts, such as a single host, may obtain the first tensor information of each device and execute step 132, and then share the obtained second tensor information with other hosts.

[0146] In step 133, the host inputs the second tensor information into the called third-party library operator, and the third-party library operator running on the host sends calculation instructions to multiple devices based on the second tensor information.

[0147] Step 134: After receiving the computation instruction, the device uses the first tensor data it stores to execute a third-party distributed computing task.

[0148] If the device already stores distributed computing tasks for third-party library operators, the computing instructions can include code to start the distributed computing tasks for third-party library operators on the device. If the device does not store distributed computing tasks for third-party library operators, the computing instructions will launch the distributed computing tasks for third-party library operators to the device so that the device can execute the distributed computing tasks for third-party library operators.

[0149] In some embodiments, in response to the output of the second tensor data after the third-party library operator performs distributed computation, the device can send the third tensor information to the host. After the host obtains the third tensor information, it converts it into the fourth tensor information. The fourth tensor information is used by the AI ​​framework to continue distributed computation.

[0150] In some embodiments, the host can also send data rearrangement requests to multiple devices. The data rearrangement requests include fourth tensor information and graph inference arrangement information. After receiving the data rearrangement request, the device rearranges the second tensor data so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

[0151] In some embodiments, the first tensor information further includes stream information corresponding to the device, and the second tensor information further includes stream information corresponding to the device. The device performs distributed computation using the first tensor data it stores based on the first distribution information and the stream information.

[0152] In some embodiments, the host can also obtain user operation information and set stream information based on the user operation information.

[0153] For the specific execution process on the host side and device side, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0154] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0155] Based on the methods in the above embodiments, this application also provides a third-party library invocation device based on a distributed system.

[0156] This application embodiment also provides a third-party library invocation device 1400 based on a distributed system. The distributed system includes at least one host and multiple devices. The device 1400 is applied to the host in the distributed system, such as... Figure 14 As shown, the device 1400 includes:

[0157] The determination module 141 is used to determine a third-party distributed computing task; wherein, the third-party distributed computing task is executed by a third-party library operator, the computing object of the third-party distributed computing task is a first tensor data, the first tensor data corresponds to a first tensor information; the first tensor information includes the meta-information of the first tensor data and the first distribution information, the first distribution information characterizes the distribution information of the first tensor data distributed in multiple devices using a first data structure supported by a deep learning framework;

[0158] The encapsulation module 142 is used to convert the first tensor information into second tensor information, wherein the second tensor information is represented by the second data structure supported by a third-party library, and the first tensor information and the second tensor information are converted using a distributed tensor memory representation data structure as the conversion medium.

[0159] Control module 143 is used to send calculation instructions to multiple devices respectively based on the second tensor information, the calculation instructions including the third-party distributed computing task, so that the multiple devices use the first tensor data stored therein to execute the third-party distributed computing task, and the multiple devices output the second tensor data after executing the third-party distributed computing task, the second tensor data being stored in the second data structure.

[0160] In some embodiments, the determining module 141 is further configured to: in response to the output of second tensor data after the plurality of said devices execute the third-party distributed computing task, obtain third tensor information, the third tensor information including: meta-information of the second tensor data and second distribution information, the second distribution information representing the distribution information of the second tensor data distributed in the plurality of said devices in the second data structure; and convert the third tensor information to obtain fourth tensor information, the fourth tensor information represented in the distributed tensor memory representation data structure, the fourth tensor information being used by the deep learning framework to continue distributed computing.

[0161] In some embodiments, after the control module 143 converts the fourth tensor information according to the third tensor information, it is further configured to: send a data rearrangement request to multiple devices, the data rearrangement request including the fourth tensor information and graph inference arrangement information, so that the devices rearrange the second tensor data after receiving the data rearrangement request, so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

[0162] In some embodiments, the first tensor information further includes stream information corresponding to the device.

[0163] In some embodiments, the device 1400 further includes an operation module for acquiring user operation information and setting stream information based on the user operation information.

[0164] In some embodiments, Figure 14 The determining module 141, encapsulation module 142, and control module 143 shown can all be implemented in software or in hardware. For example, the implementation of the determining module 141 will be described below. The implementation of other similar modules can be referenced to the implementation of the determining module 141.

[0165] As an example of a software functional unit, module 141 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 141 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0166] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0167] As an example of a hardware functional unit, module 141 may include at least one computing device, such as a server. Alternatively, module 141 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0168] The multiple computing devices included in module 141 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in module 141 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in module 141 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0169] It should be noted that, in other embodiments, the determining module 141 can be used to execute any step in the third-party library invocation method based on a distributed system described in the above embodiments, the encapsulation module 142 can be used to execute any step in the third-party library invocation method based on a distributed system described in the above embodiments, and the control module 143 can be used to execute any step in the third-party library invocation method based on a distributed system described in the above embodiments. Furthermore, the steps implemented by the determining module 141, the encapsulation module 142, and the control module 143 can be specified as needed, and different steps in the third-party library invocation method based on a distributed system described in the above embodiments can be implemented by the determining module 141, the encapsulation module 142, and the control module 143 respectively. Figure 14 The third-party library calling device 1400 based on the distributed system is shown to have all the functions.

[0170] This application also provides a computing device 1500. For example... Figure 15As shown, the computing device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate with each other via the bus 1502. The computing device 1500 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0171] Bus 1502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1504 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1504 may include a path for transmitting information between various components of the computing device 1500 (e.g., memory 1506, processor 1504, communication interface 1008).

[0172] Processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0173] The memory 1506 may include volatile memory, such as random access memory (RAM). The processor 1504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0174] The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to implement the aforementioned functions. Figure 14The functions of the determining module 141, encapsulation module 142, and control module 143 shown are configured to implement the third-party library invocation method based on a distributed system described in the above embodiments. That is, the memory 1506 stores instructions for executing the third-party library invocation method based on a distributed system described in the above embodiments.

[0175] Alternatively, the memory 1506 stores executable code, and the processor 1504 executes the executable code to implement the aforementioned functions respectively. Figure 14 The third-party library invocation device 1400 based on a distributed system shown in the figure implements the third-party library invocation method based on a distributed system described in the above embodiments. That is, the memory 1506 stores instructions for executing the third-party library invocation method based on a distributed system described in the above embodiments.

[0176] The communication interface 1503 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1500 and other devices or communication networks.

[0177] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0178] like Figure 16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the third-party library invocation method based on the distributed system described in the above embodiments.

[0179] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster may also store partial instructions for executing the third-party library invocation method based on the distributed system described in the above embodiments. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions for executing the third-party library invocation method based on the distributed system described in the above embodiments.

[0180] It should be noted that the memory 1506 in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 14 The diagram illustrates some functions of the third-party library calling device 1400 based on a distributed system. That is, the instructions stored in the memory 1506 of different computing devices 1500 can implement the functions of one or more modules among the determination module 141, encapsulation module 142, and control module 143.

[0181] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 17 One possible implementation is shown. For example... Figure 17 As shown, the two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1506 in computing device 1500A stores instructions for executing the function of determination module 141. Simultaneously, the memory 1506 in computing device 1500B stores instructions for the functions of encapsulation module 142 and control module 143.

[0182] It should be understood that Figure 17 The functions of computing device 1500A shown can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0183] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0184] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0185] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0186] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0187] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0188] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for invoking a third-party library based on a distributed system, wherein the distributed system includes at least one host and multiple devices, characterized in that, The method is applied to the host, and the method includes: A third-party distributed computing task is determined; wherein, the third-party distributed computing task is executed by a third-party library operator, the computing object of the third-party distributed computing task is a first tensor data, the first tensor data corresponds to a first tensor information; the first tensor information includes the meta-information of the first tensor data and the first distribution information, the first distribution information representing the distribution information of the first tensor data distributed in multiple devices using a first data structure supported by a deep learning framework; Based on the first tensor information, the second tensor information is obtained by conversion. The second tensor information is represented by a second data structure supported by a third-party library. The first tensor information and the second tensor information are converted using a distributed tensor memory representation data structure as the conversion medium. The third-party library operator sends calculation instructions to the multiple devices respectively based on the second tensor information. The calculation instructions include the third-party distributed computing task, so that the multiple devices use the first tensor data they store to execute the third-party distributed computing task. After executing the third-party distributed computing task, the multiple devices output the second tensor data, which is stored in the second data structure.

2. The method according to claim 1, characterized in that, The method further includes: In response to the output of second tensor data after the execution of the third-party distributed computing task by multiple devices, third tensor information is obtained. The third tensor information includes: metadata of the second tensor data and second distribution information. The second distribution information represents the distribution information of the second tensor data distributed in multiple devices in the second data structure. Based on the third tensor information, a fourth tensor information is obtained. The fourth tensor information is represented by the distributed tensor memory representation data structure and is used by the deep learning framework to continue distributed computation.

3. The method according to claim 2, characterized in that, After converting the third tensor information to obtain the fourth tensor information, the method further includes: A data rearrangement request is sent to multiple devices. The data rearrangement request includes the fourth tensor information and graph inference arrangement information, so that after receiving the data rearrangement request, the devices rearrange the second tensor data so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

4. The method according to claim 1, characterized in that, The first tensor information also includes the stream information corresponding to the device.

5. The method according to claim 4, characterized in that, The method further includes: Obtain user operation information; The stream information is set based on the user operation information.

6. A third-party library invocation device based on a distributed system, wherein the distributed system includes at least one host and multiple devices, characterized in that, The device is applied to the host computer, and the device includes: A determination module is used to determine a third-party distributed computing task; wherein, the third-party distributed computing task is executed by a third-party library operator, the computing object of the third-party distributed computing task is a first tensor data, the first tensor data corresponds to first tensor information; the first tensor information includes the meta-information of the first tensor data and the first distribution information, the first distribution information representing the distribution information of the first tensor data distributed in multiple devices using a first data structure supported by a deep learning framework; The encapsulation module is used to convert the first tensor information into second tensor information. The second tensor information is represented by a second data structure supported by a third-party library. The first tensor information and the second tensor information are converted using a distributed tensor memory representation data structure as the conversion medium. The control module is used to send calculation instructions to multiple devices respectively based on the second tensor information. The calculation instructions include the third-party distributed computing task, so that the multiple devices use the first tensor data they store to execute the third-party distributed computing task. After executing the third-party distributed computing task, the multiple devices output the second tensor data, which is stored in the second data structure.

7. The apparatus according to claim 6, characterized in that, The determining module is further configured to: In response to the output of second tensor data after the execution of the third-party distributed computing task by multiple devices, third tensor information is obtained. The third tensor information includes: metadata of the second tensor data and second distribution information. The second distribution information represents the distribution information of the second tensor data distributed in multiple devices in the second data structure. Based on the third tensor information, a fourth tensor information is obtained. The fourth tensor information is represented by the distributed tensor memory representation data structure and is used by the deep learning framework to continue distributed computation.

8. The apparatus according to claim 7, characterized in that, After the control module converts the third tensor information into the fourth tensor information, it is further configured to: A data rearrangement request is sent to multiple devices. The data rearrangement request includes the fourth tensor information and graph inference arrangement information, so that after receiving the data rearrangement request, the devices rearrange the second tensor data so that the data arrangement information of the rearranged second tensor data is consistent with the graph inference arrangement information.

9. The apparatus according to claim 6, characterized in that, The first tensor information also includes the stream information corresponding to the device.

10. The apparatus according to claim 9, characterized in that, The device further includes: an operation module; The operation module is used to acquire user operation information and set the stream information based on the user operation information.

11. A distributed system, characterized in that, The distributed system includes at least one host and multiple devices, each host or device including a processor and a memory; The processor of the at least one host is configured to execute instructions stored in the memory of the at least one host, so that the at least one host performs the method as described in any one of claims 1-5; The processor of the at least one device is configured to execute instructions stored in the memory of the at least one device, causing the plurality of devices to perform: receiving computation instructions sent by the host, and performing a distributed computation task of a third-party library operator using first tensor data stored therein according to the computation instructions.

12. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-5.

13. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-5, wherein the cluster of computing devices includes at least one computing device.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-5, wherein the computing device cluster includes at least one computing device.

Images