Model operation link establishment method, control node, cluster and program product

Abstract

The invention discloses a link establishment method for model operation, a control node, a cluster and a program product, and relates to the technical field of AI. For different communication stages of the model, the cluster determines pairs of nodes that need to perform data communications in each communication stage. The processing layer of the model is deployed in different acceleration nodes in the cluster, and no repeated acceleration node exists between different node pairs, so that the cluster can concurrently establish links for different node pairs in the same communication stage in the link establishment process of model operation, and the establishment processes of the links do not influence each other. Therefore, the problem of long link establishment time caused by serial execution of all link establishment processes can be avoided, and the link establishment efficiency of model operation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), in particular to a model running link building method, a control node, a cluster and a program product. BACKGROUND

[0002] With the development of the Internet and big data technology, AI models have made progress in information processing and knowledge extraction. The AI model (hereinafter referred to as "model" in the language of the following part) can include but is not limited to a large language model (LLM) or a model that can be used to implement a human-computer interaction function, etc. The large language model is usually deployed in a cluster, which includes a collection communication component and a plurality of computing devices, and each computing device is deployed with one or more processing layers in the large language model. In the model running process (such as model training or model inference), data communication needs to be performed between different processing layers of the model to transmit model parameters or intermediate data between the processing layers. Since different processing layers can be deployed on different computing devices, before the model runs, the collection communication component establishes full links between different computing devices to enable different processing layers deployed on different computing devices to perform data communication through the established links. However, in the model running link building process, all link establishment processes are executed in series, and the time required for model link building is relatively long, and the efficiency is relatively low. SUMMARY

[0003] The present application provides a model running link building method, a control node, a cluster and a program product, which solves the problem of long link building time caused by the serial execution of all link establishment processes in the model running link building process, and improves the efficiency of model link building.

[0004] The present application adopts the following technical solutions.

[0005] In a first aspect, the present application provides a model running link building method. The model running link building method is applied to a cluster, and the cluster includes a plurality of acceleration nodes, wherein each acceleration node includes one or more processing layers of a model. The model running link building method provided in the first aspect of the present application includes: obtaining a first request and, in response to the first request, determining a plurality of communication relationships of the plurality of acceleration nodes; wherein each communication relationship includes a plurality of node pairs of the plurality of acceleration nodes that will perform data communication in a communication stage of the model, and there is no repeated acceleration node between different node pairs. And, for a first communication relationship in the plurality of communication relationships, concurrently establishing a first group of links of the model for all node pairs in the first communication relationship. The first group of links includes a plurality of links for data communication between different processing layers of the model in the first communication stage, and one link corresponds to one node pair.

[0006] In the first optional example, the acceleration node mentioned above is an acceleration chip.

[0007] In the second optional example, the acceleration node mentioned above is an acceleration card.

[0008] In the third alternative example, the acceleration node mentioned above is an acceleration device.

[0009] In the first aspect of this application, for different communication stages of the model, the cluster determines the node pairs that need to perform data communication in each communication stage. Since the model's processing layer is deployed on different acceleration nodes in the cluster, and there are no duplicate acceleration nodes between different node pairs, the cluster can concurrently establish links for different node pairs in the same communication stage during model operation, and the link establishment process will not affect each other. This helps avoid the problem of long link establishment times caused by the serial execution of all link establishment processes, thus improving the link establishment efficiency of model operation.

[0010] In conjunction with the link-building method provided in the first aspect, in one optional implementation, the cluster also includes a set communication component. This set communication component is used to establish links for the aforementioned multiple acceleration nodes.

[0011] In conjunction with the chain-building method for model operation provided in the first aspect, in one optional implementation, the deep learning framework used by the above model is any one of the following: Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, or Theano.

[0012] In conjunction with the link-building method for model operation provided in the first aspect, in an optional implementation, after establishing the first set of links for all node pairs in the first set of communication relationships concurrently, the link-building method for model operation provided in this application further includes: for the second set of communication relationships among multiple communication relationships, establishing the second set of links for all node pairs in the second set of communication relationships concurrently. This second set of links includes multiple links for different processing layers of the model to conduct data communication in the second communication stage, and the second set of communication relationships corresponds to different communication stages than the first set of communication relationships.

[0013] In the first aspect of this application, link establishment is performed serially across different communication stages of the model to avoid disruption of communication relationships between multiple acceleration nodes. For example, establishing the first set of links first and then the second set of links helps improve the accuracy of link establishment in the model. Moreover, concurrent link establishment by different nodes within the same communication stage of the model, such as the first or second set of links mentioned above, helps reduce the link establishment time during model operation and improves the link establishment efficiency during model operation.

[0014] In conjunction with the chain-building method for model operation provided in the first aspect, in an optional implementation, the chain-building method for model operation provided in this application further includes: storing the first set of links in an established chain set. This established chain set includes: links used for data communication between different acceleration nodes in multiple communication stages of the model; each link includes: node information of the source acceleration node and node information of the target acceleration node, the node information including one or both of Internet Protocol (IP) addresses and port numbers.

[0015] In conjunction with the chain-building method for model operation provided in the first aspect, in one optional implementation, the chain-building method for model operation provided in this application further includes: applying the already established chain set to the model operation process. The model operation process includes one or more of the following: model training, model inference, or model recommendation.

[0016] In conjunction with the chain-building method for model operation provided in the first aspect, in an optional implementation, if the model is interrupted during operation, the chain-building method for model operation provided in this application further includes: saving the model's checkpoint file and obtaining the node information of the faulty acceleration node among multiple acceleration nodes. The checkpoint file includes variable information prior to the model interruption. Based on the node information of the faulty acceleration node, the cluster determines the faulty link from all links in the established chain set; this faulty link is the link associated with the faulty acceleration node. Based on the checkpoint file, the cluster determines the third set of communication relationships that have not been executed among multiple sets of communication relationships, and replaces the faulty acceleration node in all node pairs in the third set of communication relationships with a healthy node, obtaining an updated third set of communication relationships. Furthermore, based on the node pairs associated with the healthy nodes in the updated third set of communication relationships, the cluster concurrently establishes the third set of links for the model; wherein the third set of links includes: the newly created links corresponding to the node pairs associated with the healthy nodes in the third communication phase.

[0017] In the first aspect of this application, if the model is interrupted during operation, the cluster can record the variable information of the model during operation by saving the model's checkpoint file, and re-establish the fault links associated with the faulty nodes. This avoids the problem of low link establishment efficiency caused by rebuilding all links of the cluster in the usual technology, reduces the link establishment time in scenarios such as breakpoint retraining, and helps to improve the model's processing efficiency and reduce the user's waiting time.

[0018] In conjunction with the chain-building method for model operation provided in the first aspect, in an optional implementation, after the cluster concurrently establishes the third set of chains for the model, the chain-building method for model operation provided in this application further includes: configuring the state of healthy nodes based on the checkpoint file and the third set of chains; this configuration includes one or more combinations of the following: checkpoint loading, ranktable loading, and training state recovery. Furthermore, the cluster updates the set of established chains based on the third set of chains storage and applies the updated established chains to the model operation process.

[0019] Secondly, this application provides a control node. The control node includes a communication interface and a processor. The communication interface is used to obtain a first request, and the processor and the communication interface are used to collaboratively execute the operational steps of the method provided by the first aspect or any optional implementation thereof.

[0020] Thirdly, this application provides a cluster. The cluster includes a control node and multiple acceleration nodes, wherein each acceleration node includes one or more processing layers of the model. The control node is used to obtain a first request and, based on the first request, execute the operation steps of the method provided in the first aspect or any optional implementation of the first aspect to obtain multiple sets of links for the model; apply the multiple sets of links to the model's execution process, and the multiple acceleration nodes communicate data through the multiple sets of links during the model's execution process.

[0021] Fourthly, this application provides a computer program product. When the computer program product is run in a computing device, the computing device executes the operational steps of the method provided in the first aspect or any optional implementation of the first aspect. For example, the computing device may refer to the aforementioned control node or a cluster or data processing system containing a control node, etc.

[0022] The beneficial effects of the second to fourth aspects can be referred to the descriptions of any of the optional implementations of the first to fourth aspects mentioned above, and will not be repeated here. Based on the implementations provided in the above aspects, this application can also make further combinations to provide more implementations. Attached Figure Description

[0023] Figure 1 A schematic diagram of a cluster structure provided in this application Figure 1 .

[0024] Figure 2 A schematic diagram of a cluster structure provided in this application Figure 2 .

[0025] Figure 3A This application provides a topology diagram of a cluster.

[0026] Figure 3B This is a schematic diagram of distributed training of a model provided in this application.

[0027] Figure 4 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 1 .

[0028] Figure 5 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 2 .

[0029] Figure 6 The flowchart of the chain-building method for model operation provided in this application is shown in Figure 3.

[0030] Figure 7 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 4 .

[0031] Figure 8 This is a schematic diagram of the structure of a control node provided in this application. Detailed Implementation

[0032] This application provides a link establishment method for model execution. For different communication stages of the model, the cluster determines the node pairs that need to perform data communication in each stage. Since the model's processing layer is deployed on different acceleration nodes in the cluster, and there are no duplicate acceleration nodes between different node pairs, the cluster can concurrently establish links for different node pairs in the same communication stage during model execution, and the link establishment process will not affect each other. This helps avoid the problem of long link establishment times caused by the serial execution of all link establishment processes, thus improving the link establishment efficiency of model execution.

[0033] The technical solutions provided in this application can be applied not only to current model training or distributed training scenarios, but also to model inference or model recommendation scenarios, and can also be applied to future model training, distributed training, model inference, or model recommendation technologies. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The following, in conjunction with the accompanying drawings, provides exemplary descriptions of some application scenarios and system architectures that may be involved in this application.

[0034] Figure 1 A schematic diagram of a cluster structure is provided in this application, such as Figure 1 As shown, the cluster 100 includes: a control node 110 and multiple acceleration nodes, such as acceleration node i (i = 1, 2, 3...), i.e. Figure 1 Acceleration nodes 1 to 12 are shown.

[0035] The hardware implementation of control node 110 and each acceleration node will be described below with reference to the accompanying drawings.

[0036] In one alternative example, the control node 110 is an independent physical device. For example, the control node 110 can be a terminal, such as a computer, mobile terminal, tablet computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, camera, or in-vehicle computer, or it can be an edge device (e.g., a box carrying a processing chip). In this application, the control node 110 can be a computing device connected to a base station, or a computing device deployed at the base station, such as a server or cloud device.

[0037] In another alternative example, control node 110 is a control chip disposed on a physical device. This control chip can be integrated into the physical device or installed on the physical device in a pluggable manner. For example, control node 110 can be a computing chip that supports integer or floating-point calculations, such as a central processing unit (CPU), graphics processing unit (GPU) chip, neural network processing unit (NPU) chip, tensor processing unit (TPU) chip, microprocessor chip, digital signal processing (DSP), application-specific integrated circuit (ASIC) chip, or one or more integrated circuit chips used to control the execution of the computer program product provided in this application.

[0038] The two optional examples above are merely optional implementations of the control node 110 provided in the embodiments of this application, and should not be construed as limiting this application. Figure 1 In this process, different acceleration nodes communicate with control node 110. Different acceleration nodes can communicate with each other via links, which can be data communication links established by control node 110 between different acceleration nodes.

[0039] The following is about Figure 1 The hardware implementation of each acceleration node is described below.

[0040] Taking acceleration node 1 as an example, acceleration node 1 can be an acceleration device, acceleration card, or acceleration chip.

[0041] In the first feasible example, the acceleration node 1 is an acceleration device. For example, the acceleration device may include, but is not limited to, computing devices such as: hosts, servers, mobile terminals, tablets, laptops, VR devices, AR devices, MR devices, ER devices, cameras, or in-vehicle computers.

[0042] In a second feasible example, the acceleration node 1 is an acceleration card. For example, the acceleration card may include, but is not limited to, training cards, inference cards, adaptive accelerator cards, or other acceleration cards packaged with one or more chips. For instance, the acceleration card may be integrated into the rack or cabinet where cluster 100 is located, or it may be installed in a pluggable manner in the rack or cabinet where cluster 100 is located; this application does not limit this.

[0043] In a third feasible example, the acceleration node 1 is an acceleration chip. For example, this acceleration chip can include, but is not limited to, CPUs, data network processing unit (DPU) chips, GPU chips, NPU chips, TPU chips, microprocessor chips, ASIC chips, DSP chips, or other integrated circuit chips. In some alternative approaches, a single acceleration chip can include one or more dies, which can also be called a bare die, a chip, or other names. These dies can include, but are not limited to, GPUs within GPU chips, NPUs within NPU chips, or TPUs within TPU chips. For example, if a die is a GPU within a GPU chip, it can be used to perform mathematical and geometric calculations to achieve tasks such as image rendering.

[0044] Taking acceleration nodes as examples of acceleration chips, different acceleration nodes can communicate with each other using inter-chip communication links, i.e., chip-to-chip. Different acceleration chips can be connected using one or more of the following methods: High-speed custom communication system (HCCS) interface, high-speed GPU interconnect bandwidth interface, inter-integrated circuit (I2C) interface, controller area network (CAN) bus, serial peripheral interface (SPI), queued serial peripheral interface (QSPI), full-duplex asynchronous serial interface, half-duplex differential serial interface, etc. Among these, the HCCS interface is a high-speed connection channel between dies, used to accelerate data and computation to produce executable results; for example, in cluster 100, different acceleration chips use HCCS technology to connect to each other. The high-speed GPU interconnect bandwidth interface is a high-speed interconnect technology between GPUs, typically implemented through multiple pairs of wires printed on the computer board, with each pair of wires connecting to different GPUs. The I2C bus is a source-synchronous serial bus used for short-distance communication between different integrated circuits. I2C uses two lines for data transmission: a serial data line (SDL) and a serial clock line (SCL). The CAN bus is a serial communication protocol bus used for real-time applications; it can use twisted-pair cables to transmit signals. The SPI bus is a 3-wire synchronous serial full-duplex communication interface, offering advantages such as simple circuitry, high speed, and reliable communication. The QSPI bus adds a queue transmission mechanism to SPI; QSPI uses a dedicated communication interface to connect single, dual, or four data lines. A full-duplex asynchronous serial interface, also known as a universal asynchronous receiver / transmitter (UART) interface, is a universal serial data bus used for asynchronous communication. This UART bus can be a bidirectional communication bus, converting the data to be transmitted between serial and parallel communication modes; for example, a UART interface might refer to an RS-232 interface. A half-duplex differential serial interface is a serial communication bus interface that uses two wires, differential transmission, and half-duplex mode, such as the RS-485 interface.

[0045] The above description of the hardware implementation of the control node 110 and each acceleration node is merely an example provided by the embodiments of this application and should not be construed as limiting this application. In some cases, the control node 110 and each acceleration node are deployed on different physical devices. In other cases, the control node 110 and each acceleration node can be deployed on the same physical device, such as when each acceleration node is integrated or pluggable and mounted on a rack or cabinet containing the control node 110.

[0046] As an optional implementation, cluster 100 is used to train the neural network using training data until the loss function in the neural network converges and its value is less than a certain threshold, at which point the neural network training is complete, thus achieving a certain level of accuracy. Then, cluster 100 configures the trained neural network onto itself or other distributed systems. This cluster 100 or other distributed system is used to perform operations such as training, inference, or computation on requests containing input information based on the trained neural network.

[0047] In this embodiment, the aforementioned neural network can be referred to as an AI model or a model, which may refer to a large language model or other models, etc.

[0048] In one optional example, the neural network refers to a large language model (LLM). LLMs leverage their powerful computing capabilities and sophisticated algorithms to effectively process massive amounts of data, providing users with efficient and accurate information processing and analysis services. LLMs not only excel in understanding and generating human language but also demonstrate immense potential in solving complex problems and tasks. For example, LLMs are widely used in automated question-answering systems, text summarization, machine translation, and language generation, significantly improving efficiency and accuracy. Especially when dealing with large-scale datasets, LLMs can uncover deep patterns and associations, supporting decision-making. Furthermore, the self-learning ability of LLMs allows them to continuously evolve, constantly improving their performance and intelligence by learning from new data. Commonly used LLMs include the Transformer-Based Bidirectional Encoder Representation (BERT) model, which pre-trains a deep bidirectional representation using unlabeled text by jointly adjusting left and right contexts across all layers. In some optional cases, the term "large model" may also be used interchangeably with "large model." The large model provided in this application can refer not only to a large language model, but also to a model with a certain number of parameters. Depending on the domain in which the large model is applied, it can also refer to a model containing various functions such as image processing, human-computer interaction, semantic search, semantic query, or dialogue. This application does not limit the domain in which the large model can be applied or its specific name. In this document, for the sake of simplicity, it is referred to as a large model, but this should not be construed as a limitation of this application, and will not be elaborated further. Because a large language model contains many processing layers, training it using a single device takes a long time; therefore, a distributed training method is used to train the large language model.

[0049] In one alternative implementation, neural network refers to other types of networks. For example, neural network 101 is a convolutional neural network (CNN), a recurrent neural network (RNN), or a graph neural network (GNN), etc. Further implementations of CNN, RNN, or GNN can be found in the description of commonly used techniques, which will not be elaborated upon here.

[0050] For example, cluster 100 utilizes a trained neural network or model to achieve the target function intended by the neural network model, such as signage or document verification in urban commercial areas, schools, parks, and sports venues; or operations such as object detection, object recognition, or classification of data; or functions such as facial recognition payment and object classification (e.g., product classification). In addition to the target functions described in the above embodiments, neural network 101 can also achieve some functions achievable through Large Language Models (LLMs), and so on.

[0051] Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. Figure 1 The positional relationships between the devices, components, modules, etc. shown are not limiting. Depending on the user's needs for model training, cluster 100 may include more or fewer hardware components, and this application does not limit this.

[0052] This application can be applied to scenarios including but not limited to: semantic query scenarios in human-computer interaction, model training scenarios, model tuning scenarios, and scenarios involving distributed training of models. It can also be applied to model inference scenarios, model recommendation scenarios, or other scenarios.

[0053] Below Figure 1 Based on the cluster 100 shown, and taking the deployment of one or more processing layers from the model in each acceleration node as an example, the software and hardware architecture applied to the chain-building method for model operation provided in this application embodiment is introduced. Figure 2 A schematic diagram of a cluster structure provided in this application Figure 2 The software and hardware structures of this cluster 100 will be described below.

[0054] The software architecture of Cluster 100 includes training scripts and a learning framework. The training scripts include pre-defined process groups and code files corresponding to distributed data parallel computation. Pre-defined process groups consist of multiple processes, simplifying signal communication among processes within the same group; that is, if a signal is sent to a process group, all processes within that group will receive the signal. Distributed data parallel computation refers to using shared resources (such as multiple processes or multiple acceleration nodes) across multiple computers to balance the load on Cluster 100. For example, if the input data for Cluster 100 is large, it divides the input data into multiple sets, with each process or acceleration node executing only one set of computational tasks, thereby reducing the computational load on a single process or acceleration node and improving the computational efficiency of these process groups on that input data.

[0055] The learning framework includes: framework execution operators (OPs) and adapters for various collection communication libraries.

[0056] The learning framework corresponding to the framework execution operator can be one or more of the following, including but not limited to: Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, and Theano. For introductions to each learning framework, please refer to the relevant technical content; details will not be elaborated here.

[0057] The above-mentioned collection communication libraries include, but are not limited to: collection communication libraries corresponding to the Message Passing Interface (MPI), Gloo, etc. TM NCCL ( communications library), HCCL( Collective communications libraries (or other collective communications components) may be used. Different collective communications libraries have different adapters, such as... Figure 2 Shown are: MPI adapter, Gloo adapter, NCCL adapter, and HCCL adapter.

[0058] MPI is a standard or specification. Aggregate communication implemented through MPI has the following characteristics: ① Users exchange data between different nodes by explicitly sending or receiving messages. Different parallel nodes have their own independent address spaces, and access between different nodes cannot be done directly but must be achieved through explicit message passing; ② The parallel computing granularity of the cluster is large, which is suitable for large-scale scalable parallel algorithms.

[0059] Gloo TM It is an open-source library focused on collective communication, providing algorithms such as barriers, broadcasts, and all-reduce for machine learning applications. It allows the data transmission layer to flexibly switch between IP and InfiniBand, and also supports... GPUDirect technology is used to accelerate cross-node GPU-to-GPU memory transfers. This is achieved through Gloo. TM The implemented collection communication has the following characteristics: ① Data transmission is abstracted, whether in a pure IP network or an InfiniBand environment, Gloo TM Both provide good performance; ② Direct GPU access: When using InfiniBand, GPUDirect technology enables direct transfer of GPU memory, reducing unnecessary data copying and improving transfer efficiency; ③ Memory buffer processing: GlooTM Implement support for both system memory buffers and GPU buffer, no additional copying operation required.

[0060] NCCL is a high-performance multi-GPU communication library for enabling fast data transfer and collaborative computation between multiple GPUs. NCCL provides support for distributed training and data parallelism acceleration in deep learning and high-performance computing. NCCL offers processes such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive. These processes are optimized to achieve high-speed PCIe and NVLink interconnects within nodes and high bandwidth and low latency between nodes.

[0061] HCCL is a high-performance ensemble communication library based on AI processors, providing ensemble communication capabilities between single machines and multiple GPUs, as well as between multiple machines and multiple GPUs. It supports various acceleration schemes such as data parallelism, model parallelism, expert parallelism, pipeline parallelism, and sequence parallelism for large models. HCCL supports communication source domains such as allreduce, broadcast, reduce-scatter, all-gather, and all-to-all, and also supports communication algorithms such as Ring, Mesh, and Halving-Doublin. It supports ensemble communication based on links / protocols such as HCCS, RoCE (which allows remote direct memory access over Ethernet), and PCIe, and will support more links / protocols in the future.

[0062] The hardware structure of cluster 100 includes: a hardware scheduler 120, a control node 110, and multiple acceleration nodes. For example, acceleration node i (i = 1, 2, 3...). The hardware structure of cluster 100 is illustrated below.

[0063] The hardware scheduler 120 calls different collection communication libraries through the application programming interface (API) so that the collection communication component in the control node 110 can implement the chain establishment method for the model operation provided in this application.

[0064] Control node 110 includes a collective communication component for establishing links between different acceleration nodes in cluster 100. This collective communication component can be hardware or software.

[0065] In one feasible example, the aggregate communication component is hardware, such as an integrated circuit or control circuit integrated on control node 110.

[0066] In another feasible example, the collective communication component is a software program, such as an application, firmware, or software unit deployed on control node 110.

[0067] The two feasible examples above are merely optional methods for the aggregated communication component provided in the embodiments of this application and should not be construed as limiting this application. In some optional implementations, the aggregated communication component may also be independently set on a hardware device located outside the control 110 in the cluster 100, such as a backup control node, management node, or any acceleration node in the cluster 100. This application does not limit this.

[0068] Please continue reading. Figure 1 Multiple acceleration nodes, including acceleration node 1 through acceleration node 12, are connected via a mesh topology. For details on the hardware implementation of each acceleration node, please refer to [link / reference needed]. Figure 3A The description of that will not be repeated here.

[0069] The following uses a leaf-ridge network with a mesh topology as an example to illustrate the connection methods between acceleration nodes, such as... Figure 3A As shown, Figure 3A This application provides a cluster topology diagram. Figure 2 In this cluster 100, the following components are included: control node 110, spine node 1, spine node 2, leaf node 3, leaf node 4, leaf node 5, and acceleration nodes 1 through 12. For details on the implementation of control node 110, please refer to [link to relevant documentation]. Figure 3A The description of that will not be repeated here.

[0070] Please see Figure 2 Ridge node 1 connects to leaf nodes 3, 4, and 5 respectively, and ridge node 2 connects to leaf nodes 3, 4, and 5 respectively. Leaf node 3 connects to acceleration nodes 1 through 4 respectively, leaf node 4 connects to acceleration nodes 5 through 8 respectively, and leaf node 5 connects to acceleration nodes 9 through 12 respectively.

[0071] In some optional scenarios, each acceleration node may also adopt a network topology such as single-track / multi-track interconnection, which is not limited in this application.

[0072] exist Figure 3A and Figure 3B The acceleration nodes shown can be, but are not limited to, chips, acceleration cards (such as training cards and inference cards), or other acceleration devices. For example, an acceleration card refers to a pluggable card that encapsulates one or more acceleration chips. Another example is an acceleration device, which refers to a host or server equipped with one or more acceleration chips.

[0073] The following uses cluster 100 with a model deployed as an example to illustrate the model training process. It is assumed that cluster 100 in this embodiment is used to train an AI model based on the transformer architecture. The following will combine... Figure 3B One possible example is shown. Figure 3B This is a schematic diagram illustrating distributed training of a model provided in this application. Please refer to [link / reference]. Figure 1 The model includes multiple transformer modules, such as transformer module 1, transformer module 2, and transformer module n, where n is a positive integer.

[0074] Different transformer modules depend on Figure 2 Cluster 100 or Figure 3B The cluster 100 shown is used to implement this, where the attention layer corresponding to each transformer module can be trained by one or more acceleration nodes, thereby improving the training efficiency of the model. For example, different transformer modules in the model are configured to be trained on different devices. For instance, control node 110 configures transformer module 1 to acceleration node 1, transformer module 2 to acceleration node 2, and transformer module n to acceleration node 5. Other transformer modules can also be configured to other acceleration nodes in cluster 100, which will not be elaborated here.

[0075] During model training, control node 110 receives a training request and responds to it by establishing data communication links between different acceleration nodes. After control node 110 establishes links for each acceleration node, it inputs the input information corresponding to the training request into the model deployed on each acceleration node. The model's local processing layer on each acceleration node then processes the input information to perform model training.

[0076] In some optional implementations, such as when the input information is a sequence, different acceleration nodes are used to train the transformer module based on a portion of the words in the sequence and output the query results corresponding to that portion of words. It can be understood that different transformer modules in the model are trained by different acceleration nodes or chips, thus achieving parallel training of the model.

[0077] A sequence is an ordered collection of data where each element has a unique position. In some optional examples, sequences are mutable, allowing some or all elements to be modified; and sequences can be traversed iteratively, and sub-sequences can be obtained through slicing. Sequences can be used to perform one or more operations as needed, such as adding, deleting, sorting, and searching. For example, elements in a sequence can include any type of data, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues. A string is a sequence of characters, commonly used in text processing, cryptography, and image processing. An array is a sequence of elements of the same type, commonly used for storing large amounts of data and performing numerical calculations. A linked list is a sequence of nodes, commonly used to implement dynamic data structures and efficient insertion and deletion operations. A tuple is an immutable sequence, commonly used to group multiple values ​​into a single unit. A list is a mutable sequence, commonly used for storing and manipulating data. Stacks and queues are special types of sequences, commonly used to implement data structures and algorithms.

[0078] For example, a sequence can refer to a query statement, which is semantic information generated by control node 110 based on sequence data such as text or audio information. This text or audio information could be user input, transmitted from other devices, or generated by cluster 100 based on locally configured training data generation programs. See also... Figure 3B The query statement includes multiple terms (tokens), such as token1 to tokenm corresponding to the black pattern. In this document, terms (tokens) may include, but are not limited to: words, characters, punctuation marks, special symbols (such as calculation symbols), etc. In some optional cases, words may also be referred to as tokens, which is not limited in this application.

[0079] As users' demands for model processing power gradually increase, the number of words in query statements continues to grow. When using a single acceleration node to deploy the model to process long sequences, the processing efficiency of the acceleration node is relatively low. In this example, a long sequence refers to a query statement containing a large number of words. The training process of the model is briefly explained below using acceleration node 1 as an example. Control node 110 randomly or sequentially assigns the query (Q) vector, key (K) vector, and value (V) vector corresponding to different words (tokens) in the query statement to different acceleration nodes. For example, control node 110 configures the Q, K, and V corresponding to word 1 (token 1) in the query statement to acceleration node 1. Acceleration node 1 then performs multiple calculations on the key and value vectors stored locally on acceleration node 1, as well as the key and value vectors transmitted from other acceleration nodes or acceleration chips, based on the query vector stored locally on acceleration node 1. This executes the training process of transformer module 1 in the model. For example, after calculating the query vector stored locally on Accelerator Node 1 and all key-value vectors associated with the query vector in Cluster 100, Accelerator Node 1 obtains a first result, determines the loss function between the first result and the annotation of the word corresponding to the query vector stored locally on Accelerator Node 1, and finally optimizes and updates the model parameters of the attention layer 1 deployed in Accelerator Node 1 based on the loss function.

[0080] Regarding the possible implementations of the query vector (Q), key vector (K), and value vector (V), the following explanation uses a word in the query statement (such as token 1) as an example: Control node 110 performs vector transformation on token 1 to obtain the Q, K, and V corresponding to token 1. For example, the process of performing vector transformation on token 1 satisfies the following formulas (1) to (3).

[0081] Q = KW Q Formula (1)

[0082] K = XW K Formula (2)

[0083] V = XW V Formula (3)

[0084] Where X is the embedding representation of token 1, that is, the abstract representation of token 1 mapped to a low-dimensional space; Q is the query vector, W is the query vector, and W is the embedding representation of token 1. Q The weight matrix is ​​the matrix corresponding to the query vector, where K is the key vector and W is the weight matrix. K V is the weight matrix corresponding to the key vector, and W is the value vector. VThis is the weight matrix corresponding to the value vector.

[0085] In this paper, for the sake of simplicity, key vectors and value vectors can be simply referred to as key-value vectors.

[0086] In some other alternative implementations, Figure 3B The input information can also be other types of data, such as images, speech, or multimodal data. Multimodal data refers to data of different types, including text, images, video, and audio. Research on multimodal data involves processing and associating these data to achieve data fusion, aiming to provide more information for model decision-making and improve the accuracy of model processing. Furthermore, multimodal data is unstructured and heterogeneous data. When the input information is multimodal data, the model building method provided in this application can be applied to the model operation process in complex scenarios to improve the effectiveness and efficiency of model-user interaction.

[0087] Please continue reading. Figure 3B Different acceleration nodes are used to train different transformer modules in the model. The input data used to optimize and update the model parameters of different transformer modules is not entirely the same; for example, the query vectors and key-value vectors stored in different acceleration nodes are different. In other words, multiple acceleration nodes are used to train different data simultaneously, realizing a data-parallel training process for the model. Since the model's input data is a query statement, and the query statement is a sequence, data parallelism can also be called sequence parallelism. Sequence parallelism refers to dividing the model's input data into multiple sub-data and distributing different sub-data to different devices for computation, thereby reducing the computational load on a single device and improving the computational efficiency of these devices on the model's input data.

[0088] For example, transformer module 1 includes an attention layer 1 and a multilayer perceptron (MLP) 1. Attention layer 1 is used to calculate the query vector and key-value vector corresponding to input data 1 (such as token 1 and token 2 in a query statement), and finally outputs the result of attention layer 1. MLP 1 includes two linear layers, which are used to linearly process the output of attention layer 1 to obtain the output of transformer module 1 (output data 1), and update the model parameters of transformer module 1 based on this output. Figure 3B The content and training methods of other transformer modules shown can be found in the description of transformer module 1. The specific network structure of the transformer modules deployed in the acceleration nodes is described below in conjunction with...Figure 3B An example is provided.

[0089] Please see Figure 3B The transformer module consists of two parts: an encoder and a decoder.

[0090] An encoder processes input data, transforming it into an internal representation (such as an encoding matrix) that the model can understand. An encoder consists of multiple identical layers stacked on top of each other, each layer containing two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. For example... Figure 3B As shown, the encoder includes: a multi-head self-attention layer 1, an add and layer normalization (Add&Norm) operation layer (1), a feedforward layer (1), and Add&Norm (2). The multi-head self-attention layer 1 is a network processing layer determined based on the multi-head self-attention mechanism. This layer is used to calculate the dot product between the Q corresponding to a word (token) and the other K and V associated with that Q, obtaining the relevance score between each word and other words. The feedforward layer (1) is a network processing layer determined based on a feedforward neural network. This layer is used to feed forward the model parameters. Add&Norm (1) and Add&Norm (2) are used to prevent network layer degradation in the encoder and to normalize the activation values ​​of different network layers, such as by weighted summation of the relevance scores determined by the multi-head self-attention layer 1, obtaining a new representation for each word, such as the encoding matrix. In some feasible cases, a residual connection can be set before Add&Norm(1), and other types of processing layers can also be set in the encoder. This application does not limit this.

[0091] The decoder generates output data (such as output data 1) based on the encoding matrix generated by the encoder. The encoder includes: a mask multi-head self-attention layer, Add&Norm (3), multi-head self-attention layer 2, Add&Norm (4), a forward feedback layer (2), Add&Norm (5), a linear processing layer, and a normalization processing layer. The functions of Add&Norm (3) to Add&Norm (5) can be found in the description of the encoder mentioned above, and will not be repeated here. The mask multi-head self-attention layer differs from multi-head self-attention layer 2 in that: the mask multi-head self-attention layer uses a masked operation, the KV of multi-head self-attention layer 2 is calculated using the encoding matrix of the encoder, and Q is calculated using the output of the previous encoding block. The previous encoding block of multi-head self-attention layer 2 refers to the decoder corresponding to another word before predicting token 1. The linear processing layer and the normalization processing layer are used to perform linear processing and normalization operations and output the prediction result corresponding to token 1, such as output data 1.

[0092] Figure 4 The structure of the transformer model shown enables the model to handle sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0093] The link establishment process between different acceleration nodes in the model includes two stages: the first stage is the full link establishment during the initial run of the model, and the second stage is the reconstruction stage of faulty links during model operation. The link establishment process in each stage is illustrated below.

[0094] In the first stage, the model performs full chain building during its initial run. The initial run of the model can include, but is not limited to, the following scenarios: hierarchical concurrent chain building during the initialization of training / inference / recommendation tasks.

[0095] Based on the cluster 100 shown in the foregoing embodiments, the chain-building method for model operation provided in this application embodiment is described, such as... Figure 4 As shown, Figure 1 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 4 The chain-building method used by this model can be applied to the aforementioned cluster 100. Each acceleration node in cluster 100 may include one or more processing layers of the model. For a description of cluster 100, the model, and the processing layers, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0096] Please see Figure 4The chain-building method for model operation provided in this application embodiment can be executed by the control node 110. The chain-building method for model operation includes the following S401 to S404.

[0097] S401, Control Node 110 receives the first request.

[0098] In the first optional example, the first request is a model processing request, such as a model training request, a model inference request, or a model recommendation request, sent by a user device outside cluster 100 to cluster 100.

[0099] In the second alternative example, the first request is a model processing request generated by an application within cluster 100.

[0100] The two optional examples above are merely optional methods provided for the embodiments of this application and should not be construed as limiting this application. The source of the first request can also be that the user performs an operation on the cloud desktop or remote device and then sends it to cluster 100, which will not be elaborated here.

[0101] S402, Control Node 110 responds to the first request and determines multiple sets of communication relationships among multiple acceleration nodes.

[0102] Each communication relationship includes multiple pairs of nodes that will perform data communication during a communication phase of the model, with no duplicate acceleration nodes between different node pairs.

[0103] Combination Figure 4 The content shown is an illustrative example of multiple communication relationships. The multiple communication relationships in S402 may include, but are not limited to, communication relationship 1 to communication relationship 4. Communication relationship 1 may also be referred to as the first group of communication relationships, communication relationship 2 may also be referred to as the second group of communication relationships, communication relationship 3 may also be referred to as the third group of communication relationships, and communication relationship 4 may also be referred to as the fourth group of communication relationships.

[0104] For example, communication relationship 1 consists of multiple acceleration nodes. The node pairs in communication relationship 1 of the running model include: acceleration node 2 → acceleration node 1, acceleration node 4 → acceleration node 3, acceleration node 6 → acceleration node 5, and acceleration node 8 → acceleration node 7. It can be seen that there are no duplicate acceleration nodes between different node pairs; that is, the node pairs are mutually exclusive.

[0105] The node pairs in communication relationship 2 include: Accelerated Node 1 ←→ Accelerated Node 3, Accelerated Node 5 ←→ Accelerated Node 7, Accelerated Node 9 ←→ Accelerated Node 10, and Accelerated Node 11 ←→ Accelerated Node 12.

[0106] The node pairs in communication relationship 3 include: Accelerated Node 1 ←→ Accelerated Node 5, Accelerated Node 3 ←→ Accelerated Node 7, Accelerated Node 9 ←→ Accelerated Node 11, and Accelerated Node 10 ←→ Accelerated Node 12.

[0107] The node pairs in communication relationship 4 include: Accelerated Node 1 ←→ Accelerated Node 9, Accelerated Node 3 ←→ Accelerated Node 10, Accelerated Node 5 ←→ Accelerated Node 11, and Accelerated Node 7 ←→ Accelerated Node 12.

[0108] S403. For the first group of communication relationships (communication relationship 1) in multiple groups of communication relationships, the control node 110 establishes the first group of links for all nodes in the first group of communication relationships to concurrently establish models.

[0109] The first group of links comprises multiple links used for data communication between different processing layers of the model during the first communication phase, with each link corresponding to a node pair. Taking communication relationship 1 as an example, there is one link between acceleration node 2 and acceleration node 1, one link between acceleration node 4 and acceleration node 3, one link between acceleration node 6 and acceleration node 5, and one link between acceleration node 8 and acceleration node 7. Since there are no interdependent relationships between the links in the first group, the first group of links can also be referred to as the disjoint link group in the model during the first communication phase.

[0110] As can be seen from S401 to S403, for different communication stages of the model, the cluster determines the node pairs that need to perform data communication in each communication stage. Since the model's processing layer is deployed on different acceleration nodes in the cluster, and there are no duplicate acceleration nodes between different node pairs, the cluster can concurrently establish links for different node pairs in the same communication stage during model operation, and the link establishment process will not affect each other. This helps avoid the problem of long link establishment times caused by the serial execution of all link establishment processes, thus improving the link establishment efficiency of model operation.

[0111] Please continue reading. Figure 4 Following the above-described S403, the chain-building method for model operation provided in this application embodiment further includes the following S404.

[0112] S404. For the second group of communication relationships in multiple communication relationships, the control node 110 establishes a second group of links for all nodes in the second group of communication relationships to concurrently establish a model.

[0113] The second set of links includes multiple links used for data communication between different processing layers of the model in the second communication phase. The second set of communication relationships corresponds to different communication phases than the first set. Taking communication relationship 2 as an example, there is a bidirectional link between acceleration node 1 and acceleration node 3, a bidirectional link between acceleration node 5 and acceleration node 7, a bidirectional link between acceleration node 91 and acceleration node 10, and a bidirectional link between acceleration node 11 and acceleration node 12. Since there are no interdependent relationships between the links in the second set, the second set of links can also be called the disjoint link group of the model in the second communication phase.

[0114] As can be seen from S401 to S404, the links are established sequentially in different communication stages of the model to avoid disorder in the communication relationship between multiple acceleration nodes. For example, establishing the first set of links first and then the second set of links helps to improve the accuracy of the model's link establishment.

[0115] Moreover, concurrent link building between different nodes in the same communication phase of the model, such as the first or second group of links mentioned above, helps reduce the link building time during model operation and improves the link building efficiency during model operation.

[0116] To avoid redundant link establishment between node pairs with data communication needs in different communication stages, in the above... Figure 5 Based on the illustrated embodiments, this application also provides another feasible example, such as... Figure 5 As shown, Figure 2 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 5 .about Figure 5 The specific implementation of each node can be referred to the description of the aforementioned embodiments, and will not be repeated here.

[0117] Please see Figure 5 Following the above S403, the chain-building method for model operation provided in this application embodiment further includes the following S405.

[0118] S405, Control Node 110 stores the first set of links into the established link set.

[0119] The established link set includes the links used for data communication between different acceleration nodes during multiple communication phases of the model. These links include node information for the source acceleration node and the target acceleration node. The node information includes one or both of IP addresses and port numbers. The IP address refers to the network address used by the acceleration node in cluster 100, which can be used to identify the acceleration node's network location within cluster 100. The port number refers to the port (physical or logical) used by the acceleration node to connect to cluster 100, and this port number can also be used to identify the acceleration node's network location within cluster 100.

[0120] Taking the link from acceleration node 2 to acceleration node 1 in the first group of links as an example: This link includes the node information of acceleration node 2 (source acceleration node) and the node information of acceleration node 1 (target acceleration node). The node information can be represented by a combination of IP address and port number.

[0121] In one feasible example, the established chain set is stored in the memory of the control node 110.

[0122] In another feasible example, the established chain set is stored in a shared memory of cluster 100. This shared memory can support access by some or all nodes in cluster 100.

[0123] The two feasible examples above are merely optional storage methods for the established chain set provided in the embodiments of this application, and should not be construed as limiting this application. In some optional methods, the established chain set can also be stored in a database or data center that can be accessed by the control node 110. This database or data center can be located within the cluster 100 or outside the cluster 100, and this application does not limit this.

[0124] Please see Figure 5 Following the above S404, the chain-building method for model operation provided in this application embodiment further includes the following S406.

[0125] S406, Control node 110 stores the second set of links into the established link set.

[0126] Please see Figure 4 Following S405 and S406 above, the chain-building method for model operation provided in this application embodiment further includes the following S407.

[0127] S407, Control Node 110 applies the established chain set to the model's running process.

[0128] It is worth noting that the established link set used in S407 includes links of multiple acceleration nodes in all communication stages. For example, the established link set not only includes the first and second sets of links mentioned above, but also the non-intersecting link group corresponding to communication relationship 3, the non-intersecting link group corresponding to communication relationship 4, etc.

[0129] The model's operation process includes one or a combination of the following: model training, model inference, or model recommendation.

[0130] Combining the content of S401 to S407, it can be seen that since the link establishment is serial across different communication stages of the model, it is beneficial to avoid disruption of the communication relationships between multiple acceleration nodes. That is, serial link establishment between different disjoint link groups, such as establishing the first link group first and then the second link group, helps improve the accuracy of link establishment in the model. Furthermore, concurrent link establishment between different nodes within the same communication stage of the model, i.e., parallel link establishment within a single disjoint link group, such as the first or second link group mentioned above, helps reduce the link establishment time during model operation and improves the link establishment efficiency of model operation.

[0131] above Figure 5 and Figure 6 The chain-building method for model operation provided in the embodiments of this application has been described below, in conjunction with... Figure 6 The full chain building process of the model provided in this application embodiment is specifically illustrated during its first use. Figure 6 Flowchart 3 illustrates the chain-building method for a model operation provided in this application. Figure 6 In this model, the learning framework used is the PyTorch / Mindspore framework, and the set communication component in the control node 110 can be HCCL. The specific implementation of the nodes in the cluster 100 can be referred to the description in the foregoing embodiment, and will not be repeated here.

[0132] The PyTorch / Mindspore framework is used to implement training scripts for large model training tasks or inference scripts for model inference tasks. For example, the PyTorch / Mindspore framework calls HCCL to execute the set communication operator, triggering link establishment within the communication domain. A communication domain refers to a set of cooperating nodes / cards, such as accelerator node 1 to accelerator node 12. During model communication, data interaction mainly occurs between nodes within the communication domain.

[0133] Please see Figure 4 The chain-building method for model operation provided in this application includes the following S601 and S602.

[0134] The S601 and PyTorch / Mindspore frameworks execute the first communication operator call, triggering the first full connection establishment to control node 110.

[0135] For example, the PyTorch / Mindspore framework can send a first request to control node 202, as described in S401 above. For instance, after launching a model training / inference task, when the training / inference script executes the ensemble communication operator call for the first time, the PyTorch / Mindspore framework triggers the ensemble communication components MPI / Gloo / NCCL / HCCL to establish a full link within the communication domain corresponding to that ensemble communication operator.

[0136] S602, the control node 110 establishes hierarchical concurrent chains based on the set communication algorithm configured according to environment variables.

[0137] The environmental variables may include, but are not limited to, the hardware configuration of each node in cluster 100, the model's processing latency requirements, or other information. For example, the ensemble communication algorithm may refer to a model communication method executed using HCCL.

[0138] In this embodiment, hierarchical concurrent link establishment refers to serial link establishment in different communication phases and concurrent link establishment between nodes within the same communication phase, as detailed above. Figure 5 and Figure 6 The following is an example. (Referring to...) Figure 6 The optional configuration of S602 shown is illustrated by way of example. S602 may include the following S6021 to S6023k3.

[0139] S6021, Control node 110 initializes the established chain set.

[0140] For example, control node 110 can clear the information of each link stored in the established link set or migrate it to another memory. The specific implementation of the established link set can be found in the description of S405 above, and will not be repeated here.

[0141] S6022, Control node 110, according to the data transmission steps of the set communication algorithm, forms a communication relationship corresponding to a non-overlapping link group for the node pairs that have data communication in each step.

[0142] The specific implementation of S6022 can be found in the description of S402 above, and will not be repeated here.

[0143] After S6022, the control node 110 establishes links for different node pairs in the same group of communication relationships. The specific implementations of S6023k1 to S6023k3 are the same as those of S6023a1 to S6023a3. For the sake of simplicity, only S6023a1 to S6023a3 will be used as examples below.

[0144] S6023a1, Control node 110 determines whether node pair 1 has been linked.

[0145] For example, node pair 1 could be accelerated node 2 → accelerated node 1.

[0146] If node pair 1 has not established a chain, then execute S6023a2; if node pair 1 has established a chain, then execute S6023a3.

[0147] S6023a2, Control node 110 performs concurrent link establishment with other node pairs in communication relationship 1.

[0148] For details on the specific implementation of concurrent chain establishment, please refer to the descriptions in S403 or S404 above, which will not be repeated here.

[0149] S6023a3, Control node 110 stores the corresponding link of node pair 1 into the established link set.

[0150] As can be seen from S6023a1 to S6023a3, the control node 110 concurrently executes the link building process for node pairs within a non-intersecting link group, which includes: first determining whether each node pair has completed the link building process; then, for all node pairs within the non-intersecting link group that have not yet established a link, concurrently executing the link building process; and finally, after the link building is completed, placing the corresponding links of each node pair into the established link set.

[0151] In this embodiment of the application, during the execution of S602 by the control node 110, each disjoint link group is traversed serially. When all links in a disjoint link group have been established, the concurrent link establishment process corresponding to a single communication stage is repeated (such as S6023a1 to S6023a3), that is, the concurrent link establishment process in the next disjoint link group (such as S6022, S6023k1 to S6023k3) is executed until all disjoint link groups have been established, and the link establishment process of S602 ends.

[0152] It is worth noting that, Figure 4 to Figure 6 The framework illustrated above includes, but is not limited to, replacing the PyTorch / Mindspore framework with any other model framework such as TensorFlow. The collection communication libraries used include, but are not limited to, replacing HCCL with MPI / Gloo. TM / NCCL and any other collection communication library. The control node 110 of cluster 100 includes, but is not limited to, hardware forms such as GPU, NPU, DPU, TPU, and CPU. Model processing services include, but are not limited to, large model training / inference / recommendation services.

[0153] Unlike the chain-building process that relies on the master node to serially process the establishment of TCP socket / RDMA queue pair (QP) link channels for each rank, the data exchange (ranktable, IP / port, QP data, etc.) related to large model operation based on the link channels, and the disconnection of links, the chain-building method for model operation provided in this application first constructs the communication relationship corresponding to the non-overlapping link groups, and then performs chain building concurrently on the links within the non-overlapping link groups, while establishing links between non-overlapping link groups in a serial order. This reduces the model chain-building time and ensures the accuracy of model chain building.

[0154] The above Figure 7 The first stage of the model chain building process has been introduced. The second stage of the model chain building process will be illustrated below with reference to the accompanying drawings.

[0155] In the second stage, the model undergoes a fault link reconstruction process during operation.

[0156] The reasons for model failures during operation may include, but are not limited to: occasional failures of optical modules, computing nodes (such as acceleration nodes), various software (applications), or other reasons. During the complete model operation, failures often cause the training task to be interrupted, thereby triggering the training task to be restarted and training to continue from the model checkpoint saved before the interruption, i.e., breakpoint resume training.

[0157] Figure 4 A flowchart illustrating a chain-building method for model operation provided in this application. Figure 7 ,about Figure 7 The specific implementation of each node can be referred to the description of the foregoing embodiments. Based on the above embodiments, Figure 7 The cluster 100 shown also includes a fault detection module 203. This fault detection module 203 is used to perform fault detection on each acceleration node in the cluster 100.

[0158] In one feasible scenario, the fault detection module 203 is a hardware device independently located outside the control node 110, such as a processor or processing chip with fault detection function.

[0159] In another feasible scenario, the fault detection module 203 is integrated into the processing circuit of the control node 110.

[0160] In another feasible scenario, the fault detection module 203 is a software module or application deployed on the control node 110 for fault detection of each acceleration node in the cluster 100.

[0161] The above three feasible scenarios are merely optional methods for the fault detection module 203 provided in the embodiments of this application, and should not be construed as limiting this application.

[0162] Please continue reading Figure 7 If the fault detection module 203 determines that the model is interrupted during operation, the chain establishment method for model operation provided in this application embodiment includes the following S701 to S707.

[0163] S701, Control Node 110 stores the model's checkpoint file.

[0164] The checkpoint file includes variable information prior to the model's interruption. This variable information may include, but is not limited to: model parameters (such as the weights and biases of each processing layer), optimizer state (including momentum and learning rate), training state (such as the current training epoch number, batch number, etc.), and other relevant model information (such as the learning rate scheduler's state and user-defined metrics).

[0165] S702, Control Node 110 obtains node information of the faulty acceleration node among multiple acceleration nodes.

[0166] Node information may include one or both of the following: the IP address of the fault acceleration node and the port number used by the fault acceleration node.

[0167] For example, the node information of the fault acceleration node is actively reported to the control node 110 by the fault detection module 203.

[0168] For example, the node information of the fault acceleration node is fed back to the control node 110 by the fault detection module 203 after the control node 110 sends a fault query request to the fault detection module 203.

[0169] The above two examples are merely feasible ways to obtain node information of fault acceleration nodes provided in the embodiments of this application, and should not be construed as limiting this application.

[0170] S703, Control Node 110 determines the faulty link from all links in the established link set based on the node information of the faulty acceleration node.

[0171] A faulty link is a link associated with a faulty acceleration node. For example, if the faulty acceleration node is acceleration node 1, then in the first group of links corresponding to communication relationship 1, the faulty link corresponding to acceleration node 1 includes the link between acceleration node 2 and acceleration node 1; in the second group of links corresponding to communication relationship 2, the faulty link corresponding to acceleration node 1 includes the link between acceleration node 3 and acceleration node 1; in the links corresponding to communication relationship 3, the faulty link corresponding to acceleration node 1 includes the link between acceleration node 5 and acceleration node 1; and in the links corresponding to communication relationship 4, the faulty link corresponding to acceleration node 1 includes the link between acceleration node 7 and acceleration node 1.

[0172] S704, Control Node 110 determines the third communication relationship (communication relationship 3) that has not been executed among multiple communication relationships based on the checkpoint file, and replaces the faulty acceleration node in all node pairs in the third communication relationship with a healthy node to obtain the updated third communication relationship.

[0173] Understandably, control node 110 also determines that communication relationship 4 needs to be updated based on the checkpoint file, and replaces the faulty acceleration node (acceleration node 1) in communication relationship 4 with a healthy node to obtain the updated communication relationship 4.

[0174] Combination Figure 7 The term "healthy node" can refer to a standby acceleration node in cluster 100 or a newly added acceleration node to cluster 100. Figure 8 In the diagram, the healthy node is represented by a square pattern with an "H" on a white background, and the newly established link is represented by a thick dashed line after incremental link establishment.

[0175] S705, Control Node 110 associates node pairs with healthy nodes in the updated third group of communication relationships (communication relationship 3) and concurrently establishes the third group of links in the model.

[0176] The third group of links includes: newly established links between node pairs associated with healthy nodes during the third communication phase. Figure 1 (Legend of the thick dashed line after incremental link building).

[0177] After the control node 110 concurrently establishes the third set of links for the model, the link establishment method for model operation provided in this application embodiment further includes the following S706 and S707.

[0178] S706, Control Node 110 configures the status of healthy nodes based on the checkpoint file and the third group of links.

[0179] The configuration methods include one or a combination of the following: checkpoint loading, ranktalbe loading, and training state recovery.

[0180] Specifically, checkpoint loading refers to the process where the newly replaced healthy node retrieves the pre-saved checkpoint file of the model from remote storage via the network and loads the model checkpoint file into the process corresponding to that healthy node. Similarly, ranktable loading refers to the process where the control node 110 retrieves the ranktable file (such as the checkpoint file saved by S701 mentioned above) saved before the failure and loads it into the process corresponding to that healthy node. Training state recovery refers to the control node 110 restoring the various state information saved before the model training failure into the process corresponding to that healthy node.

[0181] S707, Control Node 110 updates the established chain set according to the third set of links, and applies the updated established chains to the model's operation process.

[0182] The process of S707 can be referred to the description of S407 above, and will not be repeated here.

[0183] Based on the content of S701 to S707, if the model is interrupted during operation, the cluster can record the variable information of the model during operation by saving the model's checkpoint file, and re-establish the faulty link associated with the faulty node. This avoids the problem of low link establishment efficiency caused by rebuilding all links of the cluster in the usual technology, reduces the link establishment time in scenarios such as model breakpoint retraining, and helps to improve the model's processing efficiency and reduce the user's waiting time.

[0184] It is understood that, in order to achieve the functions in the above embodiments, the cluster includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0185] The acceleration chip and cluster provided in the embodiments of this application can be referred to the description of the foregoing embodiments, and will not be repeated here. The control node provided in the embodiments of this application will be described below with reference to the accompanying drawings. Figure 8 This is a schematic diagram of a control node provided in this application. The control node 800 can be used to implement the functions of the control node in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the control node can be as follows: Figure 8 The control node 110 shown can also be a control node in subsequent embodiments, or a module (such as a chip) applied to the control node.

[0186] like Figure 8 As shown, the control node 800 may include a processor 820. Optionally, the control node 800 may also include a memory 830 and / or a communication interface 810. The processor 820 is coupled to the memory 830 and the communication interface 810, for example, via a communication bus. This communication bus may include, but is not limited to, a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0187] The following is combined with Figure 8 A detailed description of each component of control node 800 is provided below:

[0188] The processor 820 is the control center of the control node 800. It can be a single processor or a collective term for multiple processing elements. For example, the processor 820 can be one or more CPUs, an ASIC, or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0189] Optionally, the processor 820 can perform various functions of the control node 800 by running or executing software programs stored in the memory 830 and by calling data stored in the memory 830. In a specific implementation, as one embodiment, the processor 820 may include one or more CPUs.

[0190] Optionally, the control node 800 may also include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0191] The memory 830 stores the software program for executing the chain-building method of the model operation in this application, and is controlled by the processor 820 for execution. The specific implementation can be found in the chain-building method of the model operation in the above method embodiments, and will not be repeated here. For example, the memory 830 can be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 830 can be integrated with the processor 820 or exist independently, and can be controlled through the interface circuit of the control node 800 (…). Figure 8 (Not shown in the image) is coupled to the processor 820, but this application embodiment does not specifically limit this.

[0192] Communication interface 810 is used for communication with other devices. For example, if control node 800 is a user terminal (such as a client) or application server, communication interface 810 can be used to communicate with the accelerator chip or with another control node. Alternatively, if control node 800 is a multi-core chip, communication interface 810 can be used to communicate with another multi-core chip.

[0193] Optionally, the communication interface 810 may include a receiver and a transmitter. ​ (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. Optionally, the communication interface 810 can be integrated with the processor 820, or it can exist independently and be controlled via the interface circuit of the control node 800 (…). ​ (Not shown in the image) is coupled to the processor 820, but this application embodiment does not specifically limit this.

[0194] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0195] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0196] It is worth noting that if the computer program product can be provided to users through a cloud service subscription model, users can choose different subscription tiers according to their needs; for example, the computer program product can also provide enterprise-level customized services with professional domain customization, interface personalization, and extended functions according to the needs of users or enterprises. Furthermore, the computer program product provided in this application that enables the chain-building method to implement the above model can also be provided to users as a value-added service, and this application does not limit this. Alternatively, the computer program product can also be embedded in eDataMate.TM Or other toolchain systems for large language models (large models), such as operator acceleration libraries, etc.

[0197] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for chain building of model runs, characterized in that, The method is applied to a cluster comprising a plurality of acceleration nodes, wherein each acceleration node comprises one or more processing layers of a model, and the method comprises: obtaining a first request; in response to the first request, determining a plurality of communication relationships of the plurality of acceleration nodes, wherein each communication relationship comprises a plurality of node pairs of the plurality of acceleration nodes that will perform data communication in a communication stage of the model, and there is no repeated acceleration node between different node pairs; for a first communication relationship in the plurality of communication relationships, concurrently establishing a first group of links of the model for all node pairs in the first communication relationship; wherein the first group of links comprises a plurality of links between different processing layers of the model for data communication in a first communication stage, and one link corresponds to one node pair.

2. The method of claim 1, wherein, After concurrently establishing the first group of links of the model for all node pairs in the first communication relationship, the method further comprises: for a second communication relationship in the plurality of communication relationships, concurrently establishing a second group of links of the model for all node pairs in the second communication relationship, the second group of links comprising a plurality of links between different processing layers of the model for data communication in a second communication stage, and the second communication relationship corresponds to a different communication stage than the first communication relationship.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: storing the first group of links to a built link set, the built link set comprising links used for data communication between different acceleration nodes in a plurality of communication stages of the model, and the links comprising node information of a source acceleration node and node information of a target acceleration node, and the node information comprising one or both of an IP address and a port number.

4. The method of claim 3, wherein, The method further comprises: applying the built link set to a running process of the model, and the running process of the model comprising one or a combination of the following: model training, model inference, or model recommendation.

5. The method according to claim 3 or 4, characterized in that, The method further comprises: when the model running is interrupted, saving a checkpoint file of the model, and the checkpoint file comprising variable information before the model running is interrupted; obtaining node information of a faulty acceleration node in the plurality of acceleration nodes; determining a faulty link from all links in the built link set according to the node information of the faulty acceleration node, and the faulty link being a link associated with the faulty acceleration node; determining a third communication relationship in the plurality of communication relationships that is not executed according to the checkpoint file, replacing the faulty acceleration node with a healthy node in all node pairs in the third communication relationship to obtain an updated third communication relationship; concurrently establishing a third group of links of the model according to node pairs associated with the healthy node in the updated third communication relationship; wherein the third group of links comprises newly established links corresponding to a third communication stage for the node pairs associated with the healthy node.

6. The method of claim 5, wherein, After concurrently establishing the third group of links of the model, the method further comprises: According to the checkpoint file and the third set of links, the state of the health node is configured; the configuration includes one or a combination of the following: checkpoint loading, ranktable loading, training state recovery; According to the third set of links, the built chain set is updated, and the updated built chain set is applied to the running process of the model.

7. The method according to any one of claims 1 to 6, characterized in that, The cluster further includes a set communication component for establishing links for the plurality of acceleration nodes.

8. The method according to any one of claims 1 to 7, characterized in that, The deep learning framework of the model application is any one of Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, and Theano.

9. The method according to any one of claims 1-8, characterized in that, The acceleration node is an acceleration chip, an acceleration card, or an acceleration device.

10. A control node, characterized by It includes: a communication interface and a processor; The communication interface is used to obtain a first request, and the processor and the communication interface are used to cooperatively execute the method of any one of claims 1-9.

11. A cluster, characterized by It includes: a control node and a plurality of acceleration nodes, wherein each acceleration node of the plurality of acceleration nodes includes one or more processing layers of a model; The control node is configured to obtain a first request and execute the method of any one of claims 1-9 according to the first request to obtain a plurality of sets of links of the model; apply the plurality of sets of links to a running process of the model; and perform data communication through the plurality of sets of links in the running process of the model by the plurality of acceleration nodes.

12. A computer program product, characterised in that, When the computer program product is running in a computing device, the computing device executes the method of any one of claims 1-9.

Images