Link establishment method for model execution, control node, cluster, and program product

Abstract

Disclosed are a link establishment method for model execution, a control node, a cluster, and a program product, relating to the technical field of AI. For different communication phases of a model, the cluster determines node pairs that need to perform data communication in each communication phase. Since processing layers of the model are deployed across different acceleration nodes in the cluster, and no acceleration node is repeated between different node pairs, during the link establishment process for model execution, the cluster can concurrently establish links for different node pairs in the same communication phase, and the establishment of each link does not affect the others. Thus, the invention helps avoid the problem of prolonged link establishment time caused by serial execution of all link establishment processes, thereby improving the efficiency of link establishment for model execution.

Description

A method for building a link for model running, a control node, a cluster and a program product

[0001] The present application claims priority from the Chinese patent application No. 202411264685.0 filed on September 9, 2024, and entitled "A method for building a link for model running, a control node, a cluster and a program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a method for building a link for model running, a control node, a cluster and a program product. BACKGROUND

[0003] With the development of the Internet and big data technology, AI models have made progress in information processing and knowledge extraction. The AI model (hereinafter referred to as "model" in the language of the following part) can include but is not limited to a large language model (LLM) or a model that can be used to implement a human-computer interaction function, etc. The large language model is usually deployed in a cluster, which includes a collection communication component and a plurality of computing devices, and each computing device is deployed with one or more processing layers in the large language model. In the process of model running (such as model training or model inference), data communication is needed between different processing layers of the model to transmit model parameters or intermediate data between the processing layers. Since different processing layers can be deployed on different computing devices, before the model runs, the collection communication component establishes full links between different computing devices to enable different processing layers deployed on different computing devices to communicate data through the established links. However, in the process of building a link for model running, all link establishment processes are executed in series, and the time required for model link building is relatively long, and the efficiency is relatively low. SUMMARY

[0004] The present application provides a method for building a link for model running, a control node, a cluster and a program product, which solves the problem of long link building time caused by the serial execution of all link establishment processes in the process of building a link for model running, and improves the efficiency of model link building.

[0005] The present application adopts the following technical solutions.

[0006] In a first aspect, a method for building a chain for a model run is provided. The method is applied to a cluster including a plurality of acceleration nodes, each of which includes one or more processing layers of a model. The method includes obtaining a first request and, in response to the first request, determining a plurality of communication relationships of the plurality of acceleration nodes, each of which includes a plurality of node pairs of the plurality of acceleration nodes that are to perform data communication in a communication stage of the model, and there is no repeated acceleration node between different node pairs. The method further includes, for a first communication relationship of the plurality of communication relationships, concurrently establishing a first group of links of the model for all node pairs in the first communication relationship. The first group of links includes a plurality of links corresponding to the plurality of node pairs, respectively, for data communication in the first communication stage.

[0007] In a first optional example, the acceleration node is an acceleration chip.

[0008] In a second optional example, the acceleration node is an acceleration card.

[0009] In a third optional example, the acceleration node is an acceleration device.

[0010] In the first aspect, for different communication stages of the model, the cluster determines node pairs that are to perform data communication in each communication stage. Since the processing layers of the model are deployed in different acceleration nodes in the cluster, and there is no repeated acceleration node between different node pairs, the cluster can concurrently establish links for different node pairs in the same communication stage during the running of the model, and the establishment of the links does not affect each other. Thus, the method can avoid the problem of long chain building time caused by serial execution of all link establishment processes, and improve the chain building efficiency of the model run.

[0011] In an optional implementation of the method for building a chain for a model run, the cluster further includes a set communication component. The set communication component is configured to establish links for the plurality of acceleration nodes.

[0012] In an optional implementation of the method for building a chain for a model run, the deep learning framework to which the model is applied is any one of Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, and Theano.

[0013] In an optional implementation of the method for building chains for model running provided in the first aspect, after the first set of chains for the models of all node pairs in the first set of communication relationships are concurrently established, the method further includes: for a second set of communication relationships in the plurality of communication relationships, concurrently establishing a second set of chains for the models of all node pairs in the second set of communication relationships. The second set of chains includes a plurality of chains for data communication between different processing layers of the models in the second communication stage, and the second set of communication relationships corresponds to different communication stages from the first set of communication relationships.

[0014] In the first aspect of the present application, the chains are built in series for different communication stages of the model, which avoids disorder of communication relationships between the plurality of acceleration nodes, such as first establishing the first set of chains and then establishing the second set of chains, which is conducive to improving the accuracy of model chain building. Moreover, the chains are concurrently built for different node pairs in the same communication stage of the model, such as the first set of chains or the second set of chains described above, which is conducive to reducing the chain building time of model running and improving the chain building efficiency of model running.

[0015] In an optional implementation of the method for building chains for model running provided in the first aspect, the method further includes: storing the first set of chains to a built chain set. The built chain set includes chains used for data communication between different acceleration nodes in the plurality of communication stages of the model, and the chains include node information of a source acceleration node and node information of a target acceleration node, and the node information includes one or both of an internet protocol (IP) address and a port number.

[0016] In an optional implementation of the method for building chains for model running provided in the first aspect, the method further includes: applying the built chain set to a running process of the model. The running process of the model includes one or a combination of the following: model training, model inference, or model recommendation.

[0017] With reference to the method for building a chain of model running provided in the first aspect, in an optional implementation, if the model is interrupted during running, the method for building a chain of model running provided in the application further includes: saving a checkpoint file of the model, and obtaining node information of a faulty acceleration node in the plurality of acceleration nodes. The checkpoint file includes variable information before the model is interrupted. The cluster determines a faulty link from all links in the built chain set according to the node information of the faulty acceleration node, and the faulty link is a link associated with the faulty acceleration node. The cluster determines a third group of communication relationships in the plurality of groups of communication relationships that are not executed according to the checkpoint file, replaces the faulty acceleration node in all node pairs in the third group of communication relationships with a healthy node to obtain an updated third group of communication relationships, and concurrently establishes a third group of links of the model according to node pairs associated with the healthy node in the updated third group of communication relationships, wherein the third group of links includes newly built links corresponding to the third communication stage in the node pairs associated with the healthy node.

[0018] In the first aspect of the application, if the model is interrupted during running, the cluster can record the variable information of the model during running by saving the checkpoint file of the model, and re-build the faulty link associated with the faulty node, thereby avoiding the problem of low building efficiency caused by rebuilding all links of the cluster in the prior art, reducing the building time of the model in the breakpoint retraining and other scenarios, and facilitating to improve the processing efficiency of the model and reduce the waiting time of the user.

[0019] With reference to the method for building a chain of model running provided in the first aspect, in an optional implementation, after the cluster concurrently establishes the third group of links of the model, the method for building a chain of model running provided in the application further includes: configuring a state of the healthy node according to the checkpoint file and the third group of links, and the configuration includes one or a combination of the following: checkpoint loading, ranktable loading, and training state recovery. The cluster stores an updated built chain set according to the third group of links, and applies the updated built chain set to the running process of the model.

[0020] In the second aspect, the application provides a control node. The control node includes a communication interface and a processor. The communication interface is configured to obtain a first request, and the processor and the communication interface are configured to cooperatively execute the operation steps of the method provided in the first aspect or any optional implementation of the first aspect.

[0021] In a third aspect, the present application provides a cluster. The cluster comprises: a control node and a plurality of acceleration nodes, wherein each of the plurality of acceleration nodes comprises one or more processing layers of a model. The control node is configured to obtain a first request, and perform the operation steps of the method provided in the first aspect or any of the optional implementation manners of the first aspect according to the first request, to obtain a plurality of groups of links of the model; and apply the plurality of groups of links to a running process of the model, and perform data communication through the plurality of groups of links in the running process of the model by the plurality of acceleration nodes.

[0022] In a fourth aspect, the present application provides a computer program product. When the computer program product is run in a computing device, the computing device performs the operation steps of the method provided in the first aspect or any of the optional implementation manners of the first aspect. The computing device can be the control node or the cluster comprising the control node or the data processing system described above.

[0023] The beneficial effects of the second aspect to the fourth aspect can be referred to the description of the first aspect to any of the optional implementation manners of the first aspect, which will not be repeated here. On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0024] FIG. 1 is a structural schematic diagram of a cluster provided by the present application.

[0025] FIG. 2 is a structural schematic diagram of a cluster provided by the present application.

[0026] FIG. 3A is a topological structure diagram of a cluster provided by the present application.

[0027] FIG. 3B is a schematic diagram of distributed training of a model provided by the present application.

[0028] FIG. 4 is a flow schematic diagram of a link building method of model running provided by the present application.

[0029] FIG. 5 is a flow schematic diagram of a link building method of model running provided by the present application.

[0030] FIG. 6 is a flow schematic diagram of a link building method of model running provided by the present application.

[0031] FIG. 7 is a flow schematic diagram of a link building method of model running provided by the present application.

[0032] FIG. 8 is a structural schematic diagram of a control node provided by the present application. DETAILED DESCRIPTION

[0033] The application provides a method for establishing a link for model running. For different communication stages of a model, a cluster determines a node pair in each communication stage that needs to perform data communication. Since the processing layers of the model are deployed in different acceleration nodes in the cluster, and there is no repeated acceleration node between different node pairs, during the running of the model, the cluster can concurrently establish links for different node pairs in the same communication stage, and the establishment processes of the links do not affect each other. Therefore, it is beneficial to avoid the problem of long link establishment time caused by serial execution of all link establishment processes, and the link establishment efficiency of the model running is improved.

[0034] The technical solutions provided by the application can not only be applied to current model training or distributed training scenarios, but also can be applied to model inference or model recommendation scenarios, and can be applied to future model training, distributed training, model inference or model recommendation technologies. The terms used in the embodiment part of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application. Some application scenarios and system architectures that may be involved in the application are exemplarily introduced below in conjunction with the drawings.

[0035] FIG. 1 is a structural schematic diagram of a cluster provided by the application. As shown in FIG. 1, the cluster 100 includes a control node 110 and a plurality of acceleration nodes, such as acceleration node i (i = 1, 2, 3,...), that is, acceleration node 1 to acceleration node 12 shown in FIG. 1.

[0036] The hardware implementation of the control node 110 and each acceleration node will be exemplarily described below in conjunction with the drawings.

[0037] In an optional example, the control node 110 is an independent physical device. For example, the control node 110 can be a terminal such as a computer, a mobile phone terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera or a vehicle-mounted computer, and other computing devices. The control node 110 can also be an edge device (for example, a box with a processing-capable chip) and the like. In the application, the control node 110 can be a computing device connected with a base station, or a computing device deployed in the base station, such as a server or a cloud device.

[0038] In another optional example, the control node 110 is a control chip arranged on the physical device, which can be integrated on the physical device or installed on the physical device in a pluggable manner. For example, the control node 110 can be a computing chip supporting integer calculation or floating point calculation, such as a central processing unit (CPU), a graphics processing unit (GPU) chip, a neural network processing unit (NPU) chip, a tensor processing unit (TPU) chip, a micro processing chip, a digital signal processor (DSP), an application-specific integrated circuit (ASIC) chip, or one or more integrated circuit chips for controlling the computer program product provided in the scheme.

[0039] The above two optional examples are only optional implementation manners of the control node 110 provided by the embodiments of the present application, and should not be construed as a limitation of the present application. In FIG. 1, different acceleration nodes respectively communicate with the control node 110. Different acceleration nodes can communicate through a link, which can be a data communication link built by the control node 110 for different acceleration nodes.

[0040] The hardware implementation of each acceleration node shown in FIG. 1 is introduced as follows.

[0041] Taking the acceleration node 1 as an example, the acceleration node 1 can be an acceleration device, an acceleration card or an acceleration chip.

[0042] In a first feasible example, the acceleration node 1 is an acceleration device. For example, the acceleration device can include but is not limited to a host, a server, a mobile terminal, a tablet computer, a notebook computer, a VR device, an AR device, an MR device, an ER device, a camera or a vehicle-mounted computer, and the like.

[0043] In a second feasible example, the acceleration node 1 is an acceleration card. For example, the acceleration card can include but is not limited to a training card, an inference card, an adaptive accelerator card or other acceleration cards packaged with one or more chips. For example, the acceleration card can be integrated on the cabinet or rack where the cluster 100 is located, or can be installed on the cabinet or rack where the cluster 100 is located in a pluggable manner, which is not limited in the present application.

[0044] In a third possible example, the acceleration node 1 is an acceleration chip. For example, the acceleration chip can include, but is not limited to, a CPU, a data network processing unit (DPU) chip, a GPU chip, a NPU chip, a TPU chip, a microprocessing chip, an ASIC chip, a DSP chip, or other integrated circuit chips, etc. In some optional manners, a single acceleration chip can include one or more dies, which can also be referred to as a die, a wafer, a grain, or other names. For example, the die can include, but is not limited to, a GPU within a GPU chip, a NPU within a NPU chip, or a TPU within a TPU chip, etc. For example, if the die is a GPU within a GPU chip, the die can be used to perform mathematical and geometric calculations to implement image rendering and other work.

[0045] Taking an acceleration node as an acceleration chip as an example, different acceleration nodes can communicate with each other through an inter-chip communication link, that is, chip to chip. For example, different acceleration chips are connected in one or more of the following ways: a high-speed custom communication system (HCCS) interface, a high-speed GPU interconnection bandwidth interface, an inter-integrated circuit (I2C) interface, a controller area network (CAN) bus, a serial peripheral interface (SPI), a queued serial peripheral interface (QSPI), a full-duplex asynchronous serial interface, a half-duplex differential serial interface, and the like. Among them, the HCCS interface is a high-speed connection channel between dies and dies, which is used to push data and computing acceleration to obtain executable results; for example, in the cluster 100, different acceleration chips are connected in pairs using the HCCS technology. The high-speed GPU interconnection bandwidth interface is a high-speed interconnection technology between GPUs, which is usually implemented through multiple pairs of wires printed on a computer board, and the two ends of a pair of wires are connected to different GPUs. The I2C bus is a source-synchronous serial bus used for short-distance communication between different integrated circuits; I2C uses two lines for data transmission: a serial data line (SDL) and a serial clock line (SCL). The CAN bus is a serial communication protocol bus for real-time applications, which can use twisted pair wires to transmit signals. The SPI bus is a 3-wire synchronous serial full-duplex communication interface, which has the advantages of simple circuit, high speed, reliable communication, and the like. The QSPI bus is based on the SPI and adds a queue transmission mechanism; the QSPI uses a dedicated communication interface to connect single, double, or four data lines. The full-duplex asynchronous serial interface is also called a universal asynchronous receiver / transmitter (UART) interface, which is a universal serial data bus used for asynchronous communication; the UART bus can be a bidirectional communication bus, and the UART interface converts the data to be transmitted between serial communication and parallel communication, such as the RS-232 interface. The half-duplex differential serial interface is a serial communication bus interface, which adopts a two-wire system, differential transmission, and a half-duplex mode, such as the RS-485 interface.

[0046] The above description of the hardware implementation of the control node 110 and the acceleration nodes is only an example provided by the embodiments of the present application and should not be construed as a limitation of the present application. In some cases, the control node 110 and the acceleration nodes are deployed on different physical devices. In other cases, the control node 110 and the acceleration nodes can be deployed on the same physical device, such as the acceleration nodes being arranged on a rack or cabinet with the control node 110 through integration or pluggable manner.

[0047] As an optional implementation, the cluster 100 is configured to train the neural network using the training data until the loss function in the neural network converges and the loss function value is less than a certain threshold, so that the neural network reaches a certain accuracy. Further, the cluster 100 configures the trained neural network to the cluster 100 or other distributed system. The cluster 100 or other distributed system is configured to implement training, inference or calculation operations on requests containing input information according to the trained neural network.

[0048] In the present embodiment, the above-mentioned neural network can be referred to as an AI model or model, which can refer to a large language model or other model.

[0049] In an optional example, the neural network refers to a large language model (LLM). The LLM utilizes its powerful computing capabilities and complex algorithms to effectively process these massive amounts of data, providing efficient and accurate information processing and analysis services for users. The LLM not only excels in understanding and generating human language, but also exhibits great potential in solving complex problems and tasks. For example, the LLM is widely used in automated question and answer systems, text summary generation, machine translation, and language generation, greatly improving efficiency and accuracy. In particular, when dealing with large-scale data sets, the LLM can mine deep patterns and associations from them to support decision-making. In addition, the self-learning ability of the LLM enables it to continuously evolve, with its performance and intelligence constantly improving through continuous learning of new data. Commonly used LLMs include the Bidirectional Encoder Representations from Transformers (BERT) model, which can be pre-trained using unlabelled text to train deep bidirectional representations. In some optional cases, the LLM can also be referred to as a large model. The large model provided by the embodiments of the present application can not only refer to a large language model, but also refer to a model with a certain number of model parameters. For example, depending on the field in which the large model is applied, the large model can also refer to a model that includes image processing functions, human-computer interaction functions, semantic search functions, semantic query functions, or dialogue functions, among other various functions. The present application does not limit the fields in which the large model can be applied, nor the specific name of the large model. For the sake of simplicity, the large model is named in this document, but this should not be construed as a limitation of the present application, and further description will not be provided. Due to the large number of processing layers included in the LLM, training the LLM using a single device takes a long time, so a distributed training method is used to train the LLM.

[0050] In an optional implementation, the neural network refers to other types of networks. For example, the neural network 101 is a convolutional neural network (CNN), a recurrent neural network (RNN), or a graph neural network (GNN), etc. For more implementations of CNN, RNN, or GNN, please refer to the description of the general technology, which will not be described in detail herein.

[0051] For example, the cluster 100 implements a target function of a trained neural network or model, such as identification or document verification in a business district, a school, a park, a sports venue, or the like in a city, or the like, target detection, object recognition, or classification on data, or the like, face payment, object classification (such as commodity classification), or the like. In addition to the target functions of the above embodiments, the neural network 101 can also implement some functions that can be achieved by a large language model (LLM) and the like.

[0052] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, and the like shown in FIG. 1 does not constitute any limitation. According to the user's demand for model training, the cluster 100 can include more or fewer hardware components, which is not limited in the present application.

[0053] The scenarios to which the present application can be applied include, but are not limited to, a semantic query scenario of human-computer interaction, a model training scenario, a model optimization scenario, and a scenario involving distributed training of a model, and can also be applied to a model inference scenario, a model recommendation scenario, or other scenarios.

[0054] The following describes the software and hardware structures of the cluster 100 shown in FIG. 1, taking, for example, a case where one or more processing layers in a model are deployed in each acceleration node. FIG. 2 is a schematic diagram of a structure of a cluster provided by the present application. The following describes the software and hardware structures of the cluster 100, respectively.

[0055] The software structure of the cluster 100 includes a training script and a learning framework. The training script includes a preset process group and a code file corresponding to distributed data parallel computing. The preset process group includes multiple processes, and the preset process group can simplify the signal communication operation of multiple processes belonging to the same process group; that is, if a signal is sent to a process group, all processes in the process group will receive the signal. Distributed data parallel computing refers to balancing the load of the cluster 100 by using multiple computers to share resources (such as multiple processes or multiple acceleration nodes). For example, the input data of the cluster 100 is large, and the cluster 100 divides the input data into multiple groups of data, and a single process or a single acceleration node only performs a computing task corresponding to a group of data, thereby reducing the amount of computation to be performed by the single process or the single acceleration node and improving the computing efficiency of the process group on the input data.

[0056] The learning framework includes a framework execution operator (operation, OP) and an adapter corresponding to multiple collection communication libraries.

[0057] The learning framework corresponding to the framework execution operator can include one or more of the following: Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, Theano. For an introduction to each learning framework, please refer to the relevant technical content, which will not be repeated here.

[0058] The above various collective communication libraries include but are not limited to: a collective communication library corresponding to the Message Passing Interface (MPI), Gloo TM , NCCL( communications library), HCCL( collective communications library), or other collective communication components, etc. Different collective communication libraries correspond to different adapters, as shown in FIG. 2: MPI adapter, Gloo adapter, NCCL adapter and HCCL adapter.

[0059] MPI is a standard or specification representative. The collective communication implemented by MPI has the following characteristics: ① Users implement data exchange between different nodes by explicitly sending or receiving messages. Parallel different nodes have their own independent address space, and access between different nodes cannot be directly performed, and needs to be implemented through explicit message passing; ② The parallel computing granularity of the cluster is large, and is suitable for large-scale scalable parallel algorithms.

[0060] Gloo TM is an open source library focusing on collective communication, which provides algorithms including barrier, broadcast and allreduce for machine learning applications, allows the data transfer layer to flexibly switch between IP and InfiniBand, and supports GPUDirect technology to accelerate memory transfer between GPUs across nodes. The collective communication implemented by Gloo TM has the following characteristics: ① Data transfer is abstracted. Whether in a pure IP network or in an InfiniBand environment, Gloo TM can provide good performance; ② GPU direct access. When using InfiniBand, direct transfer of GPU memory is achieved through GPUDirect technology, reducing unnecessary data copying and improving transfer efficiency; ③ Memory buffer processing. Gloo TM supports both system memory buffers and GPU buffers, without the need for additional copy operations.

[0061] NCCL is a high-performance multi-GPU communication library that enables fast data transfer and collaborative computing between multiple GPUs. NCCL can provide support for distributed training and data parallel acceleration in the fields of deep learning and high-performance computing. NCCL provides all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point sending and receiving processes that are optimized for PCIe and NVLink high-speed interconnects within nodes and high-bandwidth and low-latency between nodes.

[0062] HCCL is a high-performance collective communication library based on AI processors, providing collective communication capabilities between single-machine multi-card and multi-machine multi-card. It supports various acceleration schemes such as data parallelism, model parallelism, expert parallelism, pipeline parallelism, and sequence parallelism for large models. HCCL supports communication source domains such as allreduce, broadcast, reduce-scatter, all-gather, and all to all. It also supports communication algorithms such as Ring, Mesh, and Halving-Doublin. It supports collective communication based on HCCS, RoCE (Remote Direct Memory Access over Ethernet), and PCIe links / protocols. In the future, it will support more links / protocols.

[0063] The hardware structure of the cluster 100 includes a hardware scheduler 120, a control node 110, and multiple acceleration nodes. For example, acceleration node i (i = 1, 2, 3,...), the hardware structure in the cluster 100 is exemplarily described as follows.

[0064] The hardware scheduler 120 calls different collective communication libraries through an application programming interface (API) to enable the collective communication component in the control node 110 to implement the chain establishment method for model running provided by the present application.

[0065] The control node 110 includes a collective communication component for establishing links for different acceleration nodes in the cluster 100. The collective communication component can be hardware or software.

[0066] In one possible example, the collective communication component is hardware, such as an integrated circuit or control circuit integrated on the control node 110.

[0067] In another possible example, the collective communication component is a software program, such as an application, firmware, or software unit deployed on the control node 110.

[0068] The above two feasible examples are only optional manners of the collection communication component provided by the embodiments of the present application, and should not be understood as a limitation to the present application. In some optional implementation, the collection communication component can also be independently arranged on a hardware device outside the control 110 in the cluster 100, such as a standby control node, a management node in the cluster 100, or any acceleration node in the cluster 100, which is not limited by the present application.

[0069] Please continue to refer to FIG. 2, the plurality of acceleration nodes include acceleration node 1 to acceleration node 12, and each acceleration node can be connected through a mesh topology. The hardware implementation of each acceleration node can refer to the description of FIG. 1, which is not described here.

[0070] Next, taking the mesh topology as an example, the connection mode between the acceleration nodes is described, as shown in FIG. 3A, which is a topology diagram of a cluster provided by the present application. In FIG. 3A, the cluster 100 includes: a control node 110, a spine node 1, a spine node 2, a leaf node 3, a leaf node 4, a leaf node 5, an acceleration node 1 to an acceleration node 12. The specific implementation of the control node 110 can refer to the description of FIG. 2, which is not described here.

[0071] Please refer to FIG. 3A, the spine node 1 is connected to the leaf node 3, the leaf node 4, and the leaf node 5 respectively, and the spine node 2 is connected to the leaf node 3, the leaf node 4, and the leaf node 5 respectively. The leaf node 3 is connected to the acceleration node 1 to the acceleration node 4 respectively, the leaf node 4 is connected to the acceleration node 5 to the acceleration node 8 respectively, and the leaf node 5 is connected to the acceleration node 9 to the acceleration node 12 respectively.

[0072] In some optional cases, each acceleration node can also use a single / multi-rail interconnection and other networking topologies, which are not limited by the present application.

[0073] In the contents shown in FIG. 2 and FIG. 3A, the acceleration node can be but is not limited to: a core particle, an acceleration chip, an acceleration card (such as a training card, an inference card), or other acceleration devices, etc. For example, the acceleration card refers to a pluggable card after packaging one or more acceleration chips. For another example, the acceleration device refers to a host or a server provided with one or more acceleration chips, etc.

[0074] The following describes the training process of the model by taking the deployment of the cluster 100 with the model as an example. It is assumed that the cluster 100 of the embodiments of the present application is used to train an AI model based on a transformer architecture. A possible example is shown in FIG. 3B, which is a schematic diagram of distributed training of a model provided by the present application. Referring to FIG. 3B, the model includes multiple transformer modules, such as transformer module 1, transformer module 2, and transformer module n, where n is a positive integer.

[0075] Different transformer modules are implemented by the cluster 100 in FIG. 1 or the cluster 100 shown in FIG. 2. Each attention layer corresponding to a transformer module can be trained by one or more acceleration nodes, thereby improving the training efficiency of the model. For example, different transformer modules in the model are configured to different devices for training. For example, the control node 110 configures the transformer module 1 to the acceleration node 1, the transformer module 2 to the acceleration node 2, and the transformer module n to the acceleration node 5. Other transformer modules can also be configured to other acceleration nodes in the cluster 100, which is not described here.

[0076] During the training of the model, the control node 110 obtains a training request and establishes a link for data communication between different acceleration nodes in response to the training request. After the control node 110 establishes the link for each acceleration node, the control node 110 inputs the input information corresponding to the training request to the model deployed on different acceleration nodes, and the processing layer of the model deployed locally on each acceleration node processes the input information to perform model training.

[0077] In some optional implementations, the input information is a sequence, and different acceleration nodes are used to train the transformer module according to a part of the words in the sequence, and output the query result corresponding to the part of the words. It can be understood that different transformer modules in the model are trained by different acceleration nodes or acceleration chips, which realizes the parallel training process of the model.

[0078] In some examples, the sequence is mutable, and some or all of the elements can be modified. In some examples, the sequence can be iterated through to access the elements, and the elements can be accessed by slicing. In some examples, one or more operations can be performed on the sequence, such as adding elements, deleting elements, sorting, searching, etc. For example, the elements of the sequence can include any type of elements, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues, etc. A string is a sequence of characters, which is often used in text processing, cryptography, image processing, etc. An array is a sequence of elements of the same type, which is often used to store large amounts of data and perform numerical calculations. A linked list is a sequence of nodes, which is often used to implement dynamic data structures and efficient insertion and deletion operations. A tuple is an immutable sequence, which is often used to package multiple values into a single unit. A list is a mutable sequence, which is often used to store and manipulate data. A stack and a queue are special sequences, which are often used to implement data structures and algorithms.

[0079] For example, the sequence can be a query statement, which is generated by the control node 110 based on a sequence of text or audio information, etc. The text or audio information can be input by a user, transmitted by another device, or generated by an application based on a locally configured training data generation program, etc. Referring to FIG. 3B, the query statement includes a plurality of tokens, such as token 1 to token m corresponding to the black patterns. In this context, a token can include, but is not limited to, a word, a character, a punctuation mark, a special symbol (such as a calculation symbol, etc.), etc. In some examples, a word can also be referred to as a token, which is not limited in this application.

[0080] As the user's demand for the processing capability of the model gradually increases, the number of words in the query statement increases, and the processing efficiency of the acceleration node is low when deploying the model on a single acceleration node to process long sequences. In this example, a long sequence refers to a query statement containing a large number of words. The following takes acceleration node 1 as an example to briefly describe the training process of the model. The control node 110 randomly or sequentially allocates the query (Q) vector, the key (K) vector, and the value (V) vector corresponding to different tokens in the query statement to different acceleration nodes. For example, the control node 110 configures the Q, K, and V corresponding to token 1 in the query statement to acceleration node 1, and acceleration node 1 performs multiple calculations on the key vector and the value vector stored locally by acceleration node 1 according to the query vector stored locally by acceleration node 1, the key vector and the value vector transmitted by other acceleration nodes or acceleration chips, executes the training process of the transformer module 1 in the model, and executes the training process of the transformer module 1 in the model. For example, after acceleration node 1 calculates the query vector stored locally by acceleration node 1 and all key-value vectors associated with the query vector in the cluster 100, it obtains a first result, determines the loss function between the first result and the label of the token corresponding to the query vector stored locally by acceleration node 1, and finally optimizes and updates the model parameters of the attention layer 1 deployed in acceleration node 1 according to the loss function.

[0081] For possible implementation modes of the query vector (Q), the key vector (K), and the value vector (V), the following takes a token (such as token 1) in the query statement as an example to illustrate: The control node 110 converts the token 1 into a vector to obtain the Q, K, and V corresponding to the token 1. For example, the process of converting the token 1 into a vector satisfies the following formulas (1) to (3). Q = KW Q Formula (1) K = XW K Formula (2) V = XW V Formula (3)

[0082] Wherein, X is the embedding representation corresponding to the token 1, that is, an abstract representation of mapping the token 1 to a low-dimensional space; Q is the query vector, W Q is the weight matrix corresponding to the query vector, K is the key vector, W K is the weight matrix corresponding to the key vector, V is the value vector, and W V is the weight matrix corresponding to the value vector.

[0083] In this paper, for simplicity of description, the key vector and the value vector can be referred to as the key-value vector.

[0084] In some other optional implementation manners, the input information in FIG. 3B can also be other types of data, such as images, voices, or multi-modal data, and the like. Multi-modal data refers to different types of data including text, images, videos, audio, and the like. Research on multi-modal data is a fusion problem of processing and associating these data, and the purpose is to provide more information for model decision-making and improve the accuracy of model processing. In addition, multi-modal data is unstructured and heterogeneous data. In the case where the input information is multi-modal data, the chain building method provided in the embodiments of the present application can be applied to the model running process in a complex scenario to improve the effect and efficiency of model and user interaction.

[0085] Please continue to refer to FIG. 3B. Different acceleration nodes are used to train different transformer modules in the model, and the input data used to optimize and update the model parameters of different transformer modules are not completely the same, such as the query vectors and key-value vectors stored in different acceleration nodes. That is, multiple acceleration nodes are used to train different data at the same time, realizing the data parallel training process of the model. Since the input data of the model is a query statement, and the query statement is a sequence, the data parallelism can also be called sequence parallelism. Sequence parallelism refers to: dividing the input data of the model into multiple sub-data, and distributing different sub-data to different devices for calculation, so as to reduce the amount of calculation to be performed by a single device and improve the calculation efficiency of these devices on the input data of the model.

[0086] For example, the transformer module 1 includes an attention layer 1 and a multilayer perceptron (MLP) 1. The attention layer 1 is used to calculate the query vector and the key-value vector corresponding to the input data 1 (such as token 1 and token 2 in the query statement), and finally outputs the result of the attention layer 1. The MLP 1 includes two linear layers, which are used to linearly process the result output by the attention layer 1 to obtain the output result (output data 1) of the transformer module 1, and update the model parameters of the transformer module 1 based on the output result. The contents and training methods of other transformer modules shown in FIG. 3B can refer to the description of the transformer module 1. For the specific network structure of the transformer module deployed in the acceleration node, the following will be exemplarily described in combination with FIG. 3B.

[0087] Please refer to FIG. 3B. The transformer module includes an encoder and a decoder.

[0088] The encoder is used to process the input data and convert it into an internal representation (e.g., an encoding matrix) that the model can understand. The encoder is stacked by multiple identical layers, and each layer has a network structure including two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. As shown in FIG. 3B, the encoder includes a multi-head self-attention layer 1, an Add&Norm layer (1), a feed-forward layer (1), and an Add&Norm layer (2). The multi-head self-attention layer 1 is a network processing layer determined based on a multi-head self-attention mechanism, and is configured to calculate the dot product between Q corresponding to a token and other K and V associated with the Q to obtain a correlation score between each token and other tokens. The feed-forward layer (1) is a network processing layer determined based on a feed-forward neural network, and is configured to perform forward feedback on model parameters. The Add&Norm layers (1) and (2) are configured to prevent network layer degradation in the encoder and normalize the activation values of different network layers. For example, the Add&Norm layers (1) and (2) are configured to perform weighted summation on the correlation scores determined by the multi-head self-attention layer 1 to obtain a new representation of each token, such as an encoding matrix. In some possible scenarios, a residual connection can be provided before the Add&Norm layer (1), and other types of processing layers can be provided in the encoder, which are not limited in the present application.

[0089] The decoder generates output data (e.g., output data 1) based on the encoding matrix generated by the encoder. The decoder includes a mask multi-head self-attention layer, an Add&Norm layer (3), a multi-head self-attention layer 2, an Add&Norm layer (4), a feed-forward layer (2), an Add&Norm layer (5), a linear processing layer, and a normalization processing layer. The functions of the Add&Norm layers (3) to (5) are described above in relation to the encoder, and are not repeated here. The mask multi-head self-attention layer is different from the multi-head self-attention layer 2 in that the mask multi-head self-attention layer uses a masked operation, the KV of the multi-head self-attention layer 2 is calculated using the encoding matrix of the encoder, and the Q is calculated using the output of the previous decoder block. The previous decoder block refers to a decoder corresponding to another token before the token 1 is predicted. The linear processing layer and the normalization processing layer are configured to perform linear processing and normalization operations and output a prediction result corresponding to the token 1, such as the output data 1.

[0090] The structure of the transformer model shown in FIG. 3B enables the model to process sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0091] The link establishment process between different acceleration nodes in the model includes two stages: the first stage is full link establishment of the model at the initial running, and the second stage is the re-establishment stage of the failed link during the running of the model. The model link establishment process in the two stages will be described below.

[0092] In the first stage, the full link establishment of the model at the initial running. The initial running process of the model can include but is not limited to the following scenarios: the model performs training / inference / recommendation task initialization, hierarchical and concurrent link establishment.

[0093] On the basis of the cluster 100 shown in the foregoing embodiment, the model running link establishment method provided by the embodiment of the present application is introduced, as shown in FIG. 4, which is a flowchart of a model running link establishment method provided by the present application. The model running link establishment method can be applied to the cluster 100 described above. Each acceleration node in the cluster 100 can include one or more processing layers of the model. The description of the cluster 100, the model and the processing layer can refer to the description of the foregoing embodiment, which will not be repeated here.

[0094] Referring to FIG. 4, the model running link establishment method provided by the embodiment of the present application can be executed by the control node 110. The model running link establishment method includes the following S401-S404.

[0095] S401, the control node 110 acquires a first request.

[0096] In the first optional example, the first request is a model processing request sent by a user device outside the cluster 100 to the cluster 100, such as a model training request, a model inference request or a model recommendation request.

[0097] In the second optional example, the first request is a model processing request generated by an application program in the cluster 100.

[0098] The above two optional examples are only optional ways provided by the embodiment of the present application, and should not be understood as a limitation of the present application. The source of the first request can also be a user after operating on a cloud desktop or a remote device, and sending to the cluster 100, which will not be repeated here.

[0099] S402, the control node 110 determines a plurality of groups of communication relationships of a plurality of acceleration nodes in response to the first request.

[0100] Each of the plurality of communication relations comprises a plurality of node pairs in which a plurality of acceleration nodes perform data communication in a communication stage of the model, and there is no repeated acceleration node between different node pairs.

[0101] The plurality of communication relations are exemplarily illustrated in combination with the content shown in FIG. 4. The plurality of communication relations in S402 can include but are not limited to communication relation 1 to communication relation 4. The communication relation 1 can also be referred to as the first group of communication relations, the communication relation 2 can also be referred to as the second group of communication relations, the communication relation 3 can also be referred to as the third group of communication relations, and the communication relation 4 can also be referred to as the fourth group of communication relations.

[0102] Exemplarily, the node pairs in which the plurality of acceleration nodes perform data communication in the communication relation 1 of the model include: acceleration node 2→acceleration node 1, acceleration node 4→acceleration node 3, acceleration node 6→acceleration node 5, and acceleration node 8→acceleration node 7. It can be seen that there is no repeated acceleration node between different node pairs, also referred to as each node pair being disjoint from each other.

[0103] The node pairs in the communication relation 2 include: acceleration node 1←→acceleration node 3, acceleration node 5←→acceleration node 7, acceleration node 9←→acceleration node 10, and acceleration node 11←→acceleration node 12.

[0104] The node pairs in the communication relation 3 include: acceleration node 1←→acceleration node 5, acceleration node 3←→acceleration node 7, acceleration node 9←→acceleration node 11, and acceleration node 10←→acceleration node 12.

[0105] The node pairs in the communication relation 4 include: acceleration node 1←→acceleration node 9, acceleration node 3←→acceleration node 10, acceleration node 5←→acceleration node 11, and acceleration node 7←→acceleration node 12.

[0106] S403, for the first group of communication relations (communication relation 1) in the plurality of communication relations, the control node 110 concurrently establishes a first group of links of the model for all node pairs in the first group of communication relations.

[0107] The first group of links comprises a plurality of links in which different processing layers of the model perform data communication in the first communication stage, and one link corresponds to one node pair. Taking the communication relation 1 as an example, the acceleration node 2 to the acceleration node 1 has one link, the acceleration node 4 to the acceleration node 3 has one link, the acceleration node 6 to the acceleration node 5 has one link, and the acceleration node 8 to the acceleration node 7 has one link. Since there is no mutual dependency between the links in the first group of links, the first group of links can also be referred to as a disjoint link group of the model in the first communication stage.

[0108] As can be known from S401 to S403, for different communication stages of the model, the cluster determines node pairs that need to perform data communication in each communication stage. Since the processing layers of the model are deployed in different acceleration nodes in the cluster, and there is no repeated acceleration node between different node pairs, during the running of the model, the cluster can concurrently establish links for different node pairs in the same communication stage, and the establishment processes of the links do not affect each other. Therefore, it is beneficial to avoid the problem of long link establishment time caused by serial execution of all link establishment processes, and the link establishment efficiency of the model running is improved.

[0109] Please continue to refer to FIG. 4. After S403 described above, the model running link establishment method provided in the embodiment of the application further includes the following S404.

[0110] S404, for a second group of communication relationships in the plurality of communication relationships, the control node 110 concurrently establishes a second group of links of the model for all node pairs in the second group of communication relationships.

[0111] The second group of links includes: a plurality of links in which different processing layers of the model perform data communication in the second communication stage, and the second group of communication relationships corresponds to different communication stages from the first group of communication relationships. Taking the communication relationship 2 as an example, there is a bidirectional link between the acceleration node 1 and the acceleration node 3, a bidirectional link between the acceleration node 5 and the acceleration node 7, a bidirectional link between the acceleration node 91 and the acceleration node 10, and a bidirectional link between the acceleration node 11 and the acceleration node 12. Since there is no mutual dependence relationship between the links in the second group of links, the second group of links can also be referred to as a disjoint link group of the model in the second communication stage.

[0112] As can be known from S401 to S404, the model is sequentially linked in different communication stages, and the communication relationship between the plurality of acceleration nodes is prevented from being disordered, for example, the first group of links is established first, and then the second group of links is established, which is beneficial to improve the accuracy of model link establishment.

[0113] Moreover, different node pairs in the same communication stage of the model are concurrently linked, such as the first group of links or the second group of links described above, which is beneficial to reduce the link establishment time of the model running and improve the link establishment efficiency of the model running.

[0114] In order to avoid repeated link establishment between node pairs with data communication requirements in different communication stages, on the basis of the embodiment shown in FIG. 4 described above, another feasible example provided in the embodiment of the application is shown in FIG. 5, which is a flowchart of a model running link establishment method provided in the application. The specific implementation of each node in FIG. 5 can be referred to the description of the foregoing embodiment, which will not be described here.

[0115] Referring to FIG. 5, after S403, the method for building links in model running provided by the embodiment of the present application further includes S405.

[0116] S405, the control node 110 stores the first group of links into a built link set.

[0117] The built link set includes links used by different acceleration nodes for data communication in multiple communication stages of the model. The link includes node information of a source acceleration node and node information of a target acceleration node, and the node information includes one or both of an IP address and a port number. The IP address refers to a network address used by the acceleration node in the cluster 100, and the IP address can be used to identify the network position of the acceleration node in the cluster 100. The port number refers to a port (physical port or logical port) used by the acceleration node to connect to the cluster 100, and the port number can also be used to mark the network position of the acceleration node in the cluster 100.

[0118] Taking the link from the acceleration node 2 to the acceleration node 1 in the first group of links as an example, the link includes node information of the acceleration node 2 (source acceleration node) and node information of the acceleration node 1 (target acceleration node), and the node information can be represented by a combination of an IP address and a port number.

[0119] In a feasible example, the built link set is stored in the memory of the control node 110.

[0120] In another feasible example, the built link set is stored in a shared memory of the cluster 100. The shared memory can support access by part of the nodes or all the nodes in the cluster 100.

[0121] The above two feasible examples are only optional storage manners of the built link set provided by the embodiment of the present application, and should not be understood as a limitation on the present application. In some optional manners, the built link set can also be stored in a database or a data center that can be accessed by the control node 110, and the database or the data center can be arranged inside or outside the cluster 100, which is not limited by the present application.

[0122] Referring to FIG. 5, after S404, the method for building links in model running provided by the embodiment of the present application further includes S406.

[0123] S406, the control node 110 stores the second group of links into the built link set.

[0124] Referring to FIG. 5, after S405 and S406, the method for building links in model running provided by the embodiment of the present application further includes S407.

[0125] S407, the control node 110 applies the built chain set to the running process of the model.

[0126] It is worth noting that the built chain set used in S407 includes the links of the plurality of acceleration nodes in all communication stages. The built chain set not only includes the first group of links and the second group of links described above, but also includes the disjoint link group corresponding to the communication relationship 3, the disjoint link group corresponding to the communication relationship 4, and the like.

[0127] The running process of the model includes one or a combination of the following: model training, model inference, or model recommendation.

[0128] As can be seen from the contents of S401 to S407, since the chain building of different communication stages of the model is serial, the communication relationship between the plurality of acceleration nodes is avoided from being disordered, that is, the chain building of different disjoint link groups is serial, such as first establishing the first group of links and then establishing the second group of links, which is beneficial to improve the accuracy of model chain building. In addition, the chain building of different node pairs belonging to the same communication stage of the model is concurrent, that is, the chain building of a single disjoint link group is parallel, such as the first group of links or the second group of links described above, which is beneficial to reduce the chain building time of the model running and improve the chain building efficiency of the model running.

[0129] The above FIG. 4 and FIG. 5 describe the chain building method of the model running provided by the embodiments of the present application, and the following FIG. 6 will specifically illustrate the full chain building process of the model at the initial use provided by the embodiments of the present application. FIG. 6 is a flowchart of a chain building method of a model running provided by the present application. In FIG. 6, the learning framework applied by the model is the Pytorch / Mindspore framework, and the set communication component in the control node 110 can be HCCL. The specific implementation of the nodes in the cluster 100 can refer to the description of the foregoing embodiments, and will not be repeated here.

[0130] The Pytorch / Mindspore framework is used to implement a training script for a large model training business or an inference script in a model inference task. For example, the Pytorch / Mindspore framework calls the HCCL to execute a set communication operator, and triggers the chain building in the communication domain. The communication domain refers to a set composed of a group of cooperative nodes / cards, such as the acceleration node 1 to the acceleration node 12. In the communication process of the model, data interaction mainly occurs between the nodes in the communication domain.

[0131] Referring to FIG. 6, the chain building method of the model running provided by the embodiments of the present application includes the following S601 and S602.

[0132] S601, the Pytorch / Mindspore framework performs a first communication operator call to trigger a first full chain building to the control node 110.

[0133] For example, the Pytorch / Mindspore framework can send a first request to the control node 202, as described in S401 above. For example, when the training script / inference script first executes a collective communication operator call after the model training / inference task is pulled up, the collective communication component MPI / Gloo / NCCL / HCCL triggers the full-link building for the communication domain corresponding to the collective communication operator, triggered by the Pytorch / Mindspore framework and the like.

[0134] S602, the control node 110 builds links in a hierarchical and concurrent manner according to a collective communication algorithm configured by an environment variable.

[0135] The environment variable can include, but is not limited to, the hardware configuration of each node in the cluster 100, the processing latency requirement of the model, or other information, and the like. For example, the collective communication algorithm can refer to a model communication method executed by HCCL.

[0136] In the embodiments of the present application, the hierarchical and concurrent link building refers to serial link building in different communication stages and concurrent link building of node pairs in the same communication stage. For details, refer to the foregoing embodiments of FIG. 4 and FIG. 5. The following illustrates an optional manner of S602 shown in FIG. 6, which can include the following S6021 to S6023k3.

[0137] S6021, the control node 110 initializes a built link set.

[0138] For example, the control node 110 clears or migrates the information of each link stored in the built link set to other memory. For details of the implementation of the built link set, refer to the description of S405 above, which is not repeated here.

[0139] S6022, the control node 110 groups the node pairs with data communication in each step into a communication relationship corresponding to a disjoint link group according to the data sending step of the collective communication algorithm.

[0140] For details of the implementation of S6022, refer to the description of S402 above, which is not repeated here.

[0141] After S6022, the control node 110 builds links for different node pairs in the same communication relationship, and the specific implementation of S6023k1 to S6023k3 is the same as that of S6023a1 to S6023a3. For simplicity of description, the following only exemplarily illustrates S6023a1 to S6023a3.

[0142] S6023a1, the control node 110 determines whether the node pair 1 has been built.

[0143] For example, the node pair 1 can be an accelerated node 2→an accelerated node 1.

[0144] If the node pair 1 has not been chained, S6023a2 is performed; if the node pair 1 has been chained, S6023a3 is performed.

[0145] S6023a2, the control node 110 performs concurrent chaining on the node pair 1 and other node pairs in the communication relationship 1.

[0146] The specific implementation of the concurrent chaining can refer to the description of the foregoing S403 or S404, and will not be described here.

[0147] S6023a3, the control node 110 stores the link corresponding to the node pair 1 in the chained set.

[0148] It can be known from S6023a1 to S6023a3 that the control node 110 performs the concurrent chaining process on the node pairs in one disjoint link group, including: first judging whether each node pair has completed the chaining process, starting the concurrent chaining process for all node pairs that have not been chained in the disjoint link group, and then storing the link corresponding to each node pair in the chained set after the chaining is completed.

[0149] In the embodiments of the present application, the control node 110 performs the process of S602, and serially traverses each disjoint link group. When all the links in one disjoint link group are chained, the concurrent chaining process corresponding to a single communication stage (such as S6023a1 to S6023a3) is repeated, that is, the concurrent chaining process in the next disjoint link group (such as S6022, S6023k1 to S6023k3) is performed, until all the disjoint link groups are chained and the chaining process of S602 is completed.

[0150] It is worth noting that the framework of the above process shown in FIG. 6 includes but is not limited to replacing the Pytorch / Mindspore framework with any other model framework such as tensorflow. TM The collection communication library used includes but is not limited to replacing HCCL with MPI / Gloo

[0151] Unlike relying on the master node to serially process the establishment of each rank's TCP socket / RDMA queue pair (QP) link channel, the data exchange related to the large model running based on the link channel (rank table, IP / port, QP data, etc.), and the disconnection of the link building process, the model running link building method provided by the embodiment of the application builds the communication relationship corresponding to the disjoint link group first, and then builds the link in the disjoint link group in parallel, and builds the link between the disjoint link groups in serial order, which reduces the model link building time and ensures the accuracy of the model link building.

[0152] The above Figs. 4 to 6 introduce the first stage of the model link building process, and the second stage of the model link building process is exemplarily described below in combination with the drawings.

[0153] In the second stage, the fault link reconstruction process of the model in the running process.

[0154] The reasons for the failure of the model in the running process can include but are not limited to: due to the occasional failure of optical modules, computing nodes (such as acceleration nodes), various software (application programs), etc. or other reasons, in the complete model running process, the training task is often interrupted due to failure, thereby triggering the training task to be pulled up again, and the model checkpoint saved before the interruption continues to train, i.e. breakpoint continues to train.

[0155] Fig. 7 is a flowchart of a model running link building method provided by the application, and the specific implementation of each node in Fig. 7 can be referred to the description of the foregoing embodiments. On the basis of the foregoing embodiments, the cluster 100 shown in Fig. 7 further includes a fault detection module 203. The fault detection module 203 is configured to detect the fault of each acceleration node in the cluster 100.

[0156] In a feasible case, the fault detection module 203 is a hardware device independently arranged outside the control node 110, such as a processor or a processing chip with a fault detection function.

[0157] In another feasible case, the fault detection module 203 is integrated in the processing circuit of the control node 110.

[0158] In yet another feasible case, the fault detection module 203 is a software module or an application program deployed in the control node 110, and is configured to detect the fault of each acceleration node in the cluster 100.

[0159] The above three feasible cases are only optional ways of the fault detection module 203 provided by the embodiments of the application, and should not be understood as a limitation of the application.

[0160] Please continue to refer to FIG. 7, if the fault detection module 203 determines that the model is interrupted during running, the model running chain building method provided by the embodiments of the present application includes the following S701 to S707.

[0161] S701, the control node 110 saves the checkpoint file of the model.

[0162] The checkpoint file includes variable information before the model is interrupted. The variable information can include but is not limited to: parameters of the model (such as weights and biases of each processing layer), optimizer state (including momentum, learning rate), training state (such as the current training round, batch number, etc.), and other related model information (such as the state of the learning rate scheduler, user-defined indicators), etc.

[0163] S702, the control node 110 obtains the node information of the fault acceleration node in the plurality of acceleration nodes.

[0164] The node information can include one or both of the IP address of the fault acceleration node and the port number used by the fault acceleration node.

[0165] For example, the node information of the fault acceleration node is actively reported by the fault detection module 203 to the control node 110.

[0166] For another example, the node information of the fault acceleration node is fed back by the fault detection module 203 to the control node 110 after the control node 110 sends a fault query request to the fault detection module 203.

[0167] The above two examples are only feasible ways to obtain the node information of the fault acceleration node provided by the embodiments of the present application, and should not be understood as a limitation of the present application.

[0168] S703, the control node 110 determines the fault link from all links in the built chain set according to the node information of the fault acceleration node.

[0169] The fault link is the link associated with the fault acceleration node. For example, if the fault acceleration node is acceleration node 1, the fault link corresponding to the acceleration node 1 in the first group of links corresponding to the communication relationship 1 includes the link between the acceleration node 2 and the acceleration node 1; the fault link corresponding to the acceleration node 1 in the second group of links corresponding to the communication relationship 2 includes the link between the acceleration node 3 and the acceleration node 1; the fault link corresponding to the acceleration node 1 in the link corresponding to the communication relationship 3 includes the link between the acceleration node 5 and the acceleration node 1; the fault link corresponding to the acceleration node 1 in the link corresponding to the communication relationship 4 includes the link between the acceleration node 7 and the acceleration node 1.

[0170] S704, the control node 110 determines a third group of communication relationships (communication relationship 3) in the plurality of groups of communication relationships that are not executed according to the checkpoint file, replaces the failed acceleration node in all node pairs in the third group of communication relationships with a healthy node, and obtains an updated third group of communication relationships.

[0171] It can be understood that the control node 110 also determines that communication relationship 4 needs to be updated according to the checkpoint file, and replaces the failed acceleration node (acceleration node 1) in communication relationship 4 with a healthy node to obtain an updated communication relationship 4.

[0172] The healthy node can be a backup acceleration node in the cluster 100 or a new acceleration node accessing the cluster 100. In FIG. 7, the healthy node is a square pattern with a white background "H", and the new link is a thick dashed line legend after incremental link building.

[0173] S705, the control node 110 concurrently establishes a third group of links of the model according to the healthy node associated node pairs in the updated third group of communication relationships (communication relationship 3).

[0174] The third group of links includes new links (thick dashed line legend after incremental link building in FIG. 7) of the healthy node associated node pairs in the third communication phase.

[0175] After the control node 110 concurrently establishes the third group of links of the model, the model running link building method provided by the embodiment of the application further includes the following S706 and S707.

[0176] S706, the control node 110 configures the state of the healthy node according to the checkpoint file and the third group of links.

[0177] The configuration method includes one or a combination of the following methods: checkpoint loading, ranktable loading, and training state recovery.

[0178] The checkpoint loading means that the newly replaced healthy node obtains the pre-saved checkpoint file of the model from a remote storage through a network, and loads the model checkpoint file into the process corresponding to the healthy node. The ranktable loading means that the ranktable file saved before the failure (such as the checkpoint file saved in S701) is obtained from the control node 110 and loaded into the process corresponding to the healthy node. The training state recovery means that the control node 110 recovers various state information saved before the model training failure to the process corresponding to the healthy node.

[0179] S707. The control node 110 stores the updated set of built chains according to the third set of links, and applies the updated set of built chains to the running of the model.

[0180] The process of S707 can refer to the foregoing description of S407, and will not be described here.

[0181] In combination with the content of S701 to S707, if the model is interrupted during running, the cluster can record the variable information of the model during running by saving the checkpoint file of the model, and rebuild the fault links associated with the fault node, thereby avoiding the problem of low efficiency of chain building caused by rebuilding all links of the cluster in the prior art, reducing the chain building time in the breakpoint retraining and other scenarios of the model, and being beneficial to improving the processing efficiency of the model and reducing the waiting time of the user.

[0182] It can be understood that, in order to realize the functions in the above embodiments, the cluster includes the hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0183] The acceleration chip and the cluster provided in the embodiments of the present application can refer to the foregoing description of the embodiments, and will not be described here. Next, the control node provided in the embodiments of the present application will be introduced in combination with the accompanying drawings. FIG. 8 is a structural schematic diagram of a control node provided in the present application. The control node 800 can be used to realize the functions of the control node in the foregoing method embodiments, and thus can also realize the beneficial effects possessed by the foregoing method embodiments. In the present embodiment, the control node can be the control node 110 as shown in FIG. 1, and can also be the control node in the subsequent embodiments, or a module (such as a chip) applied to the control node.

[0184] As shown in FIG. 8, the control node 800 can include a processor 820. Optionally, the control node 800 can also include a memory 830 and / or a communication interface 810. The processor 820 is coupled with the memory 830 and the communication interface 810, which can be connected through a communication bus. The communication bus can include, but is not limited to, a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0185] The various components of the control node 800 will be described below in conjunction with FIG. 8:

[0186] The processor 820 is the control center of the control node 800, and can be one processor or a collective term of multiple processing elements. For example, the processor 820 can be one or more CPUs, or can be an ASIC, or can be one or more integrated circuits configured to implement one or more embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGA).

[0187] Optionally, the processor 820 can perform various functions of the control node 800 by running or executing software programs stored in the memory 830, and invoking data stored in the memory 830. In a specific implementation, as an embodiment, the processor 820 can include one or more CPUs.

[0188] Optionally, the control node 800 can also include multiple processors. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0189] The memory 830 is configured to store a software program for performing the chain building method of the model running in the scheme of the present application, and the processor 820 is configured to control the execution of the software program. For the specific implementation, refer to the chain building method of the model running in the method embodiments described above, which will not be repeated here. For example, the memory 830 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disc storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but the present application is not limited thereto. The memory 830 can be integrated with the processor 820 or exist independently and coupled with the processor 820 through the interface circuit (not shown in FIG. 8) of the control node 800, and the present application is not limited in this regard.

[0190] The communication interface 810 is configured to communicate with other devices. For example, the control node 800 is a client or an application server, and the communication interface 810 can be used to communicate with an acceleration chip or another control node. For another example, the control node 800 is a multi-core chip, and the communication interface 810 can be used to communicate with another multi-core chip.

[0191] Optionally, the communication interface 810 can include a receiver and a transmitter (not shown separately in FIG. 8). The receiver is configured to implement the receiving function, and the transmitter is configured to implement the transmitting function. Optionally, the communication interface 810 can be integrated with the processor 820 or exist independently and coupled with the processor 820 through the interface circuit (not shown in FIG. 8) of the control node 800, and the present application is not limited in this regard.

[0192] The method steps in the embodiments can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a RAM, a flash memory, a ROM, a PROM, an EPROM, an EEPROM, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0193] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable devices. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired or wireless manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (DVD); or a semiconductor medium, such as a solid state drive (SSD).

[0194] It is worth noting that if the computer program product can be provided to users for use through a cloud service subscription mode, users can choose different subscription levels according to their needs; for example, the computer program product can also provide enterprise-level customized services with professional domain customization, interface personalization, and expansion functions according to the needs of users or enterprises. In addition, the computer program product provided by the present application, which can implement the chain building method of the above model, can also be made into a value-added service to provide users, which is not limited by the present application. Alternatively, the computer program product can also be embedded in eDataMateTM or other large language model (large model) toolchain systems, such as operator acceleration libraries, and the like.

[0195] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A chain-building method for model operation, characterized in that, The method is applied to a cluster comprising multiple acceleration nodes, wherein each acceleration node comprises one or more processing layers of the model, and the method includes: Get the first request; In response to the first request, multiple sets of communication relationships of the plurality of acceleration nodes are determined, wherein each set of communication relationships includes: multiple node pairs in which the plurality of acceleration nodes will perform data communication in a communication phase of the model, and there are no duplicate acceleration nodes between different node pairs; For the first group of communication relationships in the multiple groups of communication relationships, the first group of links of the model is established concurrently for all node pairs in the first group of communication relationships; wherein, the first group of links includes: multiple links for data communication between different processing layers of the model in the first communication stage, and one link corresponds to one node pair.

2. The method according to claim 1, characterized in that, After establishing the first set of links of the model concurrently for all node pairs in the first set of communication relationships, the method further includes: For the second group of communication relationships among the multiple communication relationships, a second group of links of the model is established concurrently for all node pairs in the second group of communication relationships. The second group of links includes multiple links in which different processing layers of the model communicate data in the second communication stage. The second group of communication relationships corresponds to a different communication stage than the first group of communication relationships.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The first set of links is stored in the established link set, which includes: the links used for data communication between different acceleration nodes in multiple communication stages of the model, and the links include: node information of the source acceleration node and node information of the target acceleration node, wherein the node information includes one or both of IP address and port number.

4. The method according to claim 3, characterized in that, The method further includes: The established chain set is applied to the model's operation process, which includes one or a combination of the following: model training, model inference, or model recommendation.

5. The method according to claim 3 or 4, characterized in that, The method further includes: When the model operation is interrupted, the checkpoint file of the model is saved. The checkpoint file includes variable information before the model operation was interrupted. Obtain the node information of the faulty acceleration node among the multiple acceleration nodes; Based on the node information of the fault acceleration node, the faulty link is determined from all links in the established link set; the faulty link is the link associated with the fault acceleration node. Based on the checkpoint file, determine the third group of communication relationships that has not been executed among the multiple groups of communication relationships, and replace the faulty acceleration node in all node pairs in the third group of communication relationships with a healthy node to obtain the updated third group of communication relationships. Based on the node pairs associated with the healthy node in the updated third group of communication relationships, the third group of links of the model is established concurrently; wherein, the third group of links includes: the newly established links corresponding to the node pairs associated with the healthy node in the third communication phase.

6. The method according to claim 5, characterized in that, After the third set of links of the model are established concurrently, the method further includes: Configure the state of the healthy node according to the checkpoint file and the third set of links; the configuration method includes one or a combination of the following: checkpoint loading, ranktalbe loading, training state recovery; The established chain set is updated according to the third set of links, and the updated established chains are combined and applied to the operation of the model.

7. The method according to any one of claims 1-6, characterized in that, The cluster also includes a collective communication component, which is used to establish links for the plurality of acceleration nodes.

8. The method according to any one of claims 1-7, characterized in that, The deep learning framework used by the model is any one of the following: Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, or Theano.

9. The method according to any one of claims 1-8, characterized in that, The acceleration node is an acceleration chip, acceleration card, or acceleration device.

10. A control node, characterized in that, include: Communication interface and processor; The communication interface is used to obtain a first request, and the processor and the communication interface are used to collaboratively execute the method of any one of claims 1-9.

11. A cluster, characterized in that, include: A control node and multiple acceleration nodes, wherein each of the multiple acceleration nodes includes one or more processing layers of the model; The control node is used to obtain a first request and execute the method of any one of claims 1-9 according to the first request to obtain multiple sets of links of the model; apply the multiple sets of links to the running process of the model, and have the multiple acceleration nodes perform data communication through the multiple sets of links during the running process of the model.

12. A computer program product, characterized in that, When the computer program product is run in a computing device, the computing device performs the method of any one of claims 1-9.

Images