Optimization Methods for Parallel Training and Inference Adaptation of Next-Generation Heterogeneous Supercomputing Large Models

Abstract

This application relates to a method for parallel training and inference adaptation optimization of large models in next-generation heterogeneous supercomputing systems. It obtains model context relationships and inter-layer tensor dependencies, constructs a directed acyclic dependency graph based on these relationships, and encapsulates operators within the graph to obtain an iterative execution queue. Communication modules are allocated to each computing node in the parallel computing process, and each node executes corresponding batch iterative tasks according to the iterative execution queue. During the execution of the iterative execution queue, each computing node starts an operator engine within its allocated communication module, splits and rearranges tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed. The computation results of each computing node are collected to obtain the iterative task computation results. By employing a batch partitioning method to divide the iterative batch tasks into multiple computing nodes for parallel computation, efficient data processing is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models. Background Technology

[0002] In today's digital age, large-scale model training and inference have become key technologies driving the development of various industries. They are widely used in fields such as finance, healthcare, education, autonomous driving, and intelligent cockpit systems, greatly enhancing the level of intelligence in these industries by providing personalized services, optimizing decision-making processes, and improving user experience.

[0003] Most current terminal computing devices, such as NVIDIA GPUs, AMD GPUs, Intel CPUs, and Google TPUs, which have the computing power and storage required to guarantee the training and inference of large models, need to have a high-performance computing library that supports both the model data flow level and the underlying computing level to support the large number of floating-point operations and huge memory overhead generated by the training and inference of large models.

[0004] The batch processing requirements for training and inference of large models consume significant computing resources and memory, often exceeding the capacity of a single machine. Therefore, achieving efficient training and inference of large models has become a pressing technical challenge. Summary of the Invention

[0005] Therefore, it is necessary to provide an efficient method, apparatus, computer equipment, computer-readable storage medium, and computer program product for parallel training and inference adaptation and optimization of next-generation heterogeneous supercomputing large models, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models. The method includes:

[0007] Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue;

[0008] The communication module is assigned to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative task according to the iterative execution queue.

[0009] During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed.

[0010] The calculation results of each computing node are collected to obtain the calculation results of the iterative task.

[0011] In one embodiment, the steps of obtaining model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue include:

[0012] Obtain model training and inference task requests;

[0013] In response to the model training and inference task request, obtain the model context information;

[0014] Obtain the tensor dependencies between model layers, and based on the model context information and the tensor dependencies between model layers, obtain the computation operators used to describe the tensor computation mode in this operation.

[0015] For each tensor, a tensor description is used as a graph node. Each graph node encapsulates a tensor data type, a computation operator, a pointer to a gradient tensor descriptor, and a pointer to a source tensor descriptor.

[0016] Extract all source tensor identifiers pointed to by the tensor identifiers in the final layer of the model to obtain the target source tensor identifier;

[0017] Based on the target source tensor identifier, all tensor corresponding operators of the model are reverse-connected to generate a directed acyclic depth dependency graph;

[0018] Using the final layer tensor of the model as the initial node, a depth-first traversal is used to obtain the iterative execution queue.

[0019] In one embodiment, allocating the communication module to each computing node in parallel computing includes:

[0020] Obtain the preset number of batch tasks corresponding to model training and inference, and the total number of computing nodes N activated;

[0021] Based on the preset number of batch tasks and the total number of computing nodes N, a communication block is allocated to each computing node. The communication module includes a single main block and multiple auxiliary blocks.

[0022] In one embodiment, before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the method further includes:

[0023] Obtain the model training text corresponding to the training task;

[0024] Based on the model training file, determine the number of training sampling batches and the number of iterations for each iteration;

[0025] The steps of obtaining model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on these relationships, and encapsulating operators according to the graph to obtain an iterative execution queue include:

[0026] Obtain the model context relationships and inter-layer tensor dependencies for the training task;

[0027] Based on the model context relationships and inter-layer tensor dependencies, a forward dependency graph is constructed;

[0028] Based on the tensor dependency relationship calculated by gradient, construct the inverse dependency graph;

[0029] The forward dependency graph and the reverse dependency graph are concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

[0030] In one embodiment, the allocation of the communication module to each computing node in the parallel computing, and the execution of corresponding batch iterative tasks by each computing node according to the iterative execution queue, includes:

[0031] The communication module is assigned to each computing node in the parallel computing process, and the batch data of the training task is obtained.

[0032] The batch data is evenly distributed to the communication modules corresponding to each computing node for computation.

[0033] Each computing node performs the calculation of the forward propagation tensor and the backward propagation gradient tensor according to the iterative execution queue;

[0034] The calculation results of each of the computing nodes are collected to obtain the calculation results of the iterative task, including:

[0035] The calculation results of the communication modules in each computing node are collected, and an all-reduce operation is performed on the calculation results to synchronize global gradient data and obtain the calculation results of the iterative task.

[0036] In one embodiment, before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the method further includes:

[0037] Receive prompts required for model inference;

[0038] Set the number of batches of text sequences that need to be generated through inference;

[0039] Based on the prompt statement and the number of batches to be generated according to the text sequence, generate the text sequence;

[0040] The allocation of communication modules to each computing node in parallel computing, and the execution of corresponding batch iterative tasks by each computing node according to the iterative execution queue, includes:

[0041] The communication module is assigned to each computing node in the parallel computing process, and the number of iterations required is determined by the communication module according to the length that the text sequence needs to be inferred and generated.

[0042] Based on the required number of iterations, the communication module performs the corresponding batch iteration tasks in the iteration execution queue.

[0043] In one embodiment, the process of aggregating the computation results of each computing node to obtain the iterative task computation result includes:

[0044] The calculation results of each computing node are collected, and the text sequences generated by each computing node are extracted from the calculation results of each computing node.

[0045] Record the time it takes for each computing node to complete the inference generation task and the text sequence it generates.

[0046] In one embodiment, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit, and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0047] In one embodiment, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block by the main component through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

[0048] Secondly, this application also provides a device for parallel training and inference adaptation and optimization of large models in next-generation heterogeneous supercomputing systems. The device includes:

[0049] The queue generation module is used to obtain the model context relationship and the inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue.

[0050] The task allocation module is used to allocate the communication module to each computing node in the parallel computing, and each computing node executes the corresponding batch iterative tasks according to the iterative execution queue.

[0051] The parallel iteration module is used to enable each computing node to start the operator engine within the allocated communication module during the execution of the iteration execution queue, split and rearrange the tensor data, and perform parallel tensor computation within the computing node through the communication module until the data in the iteration execution queue is processed.

[0052] The data collection module is used to collect the calculation results of each computing node to obtain the calculation results of the iterative task.

[0053] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0054] Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue;

[0055] The communication module is assigned to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative task according to the iterative execution queue.

[0056] During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed.

[0057] The calculation results of each computing node are collected to obtain the calculation results of the iterative task.

[0058] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0059] Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue;

[0060] The communication module is assigned to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative task according to the iterative execution queue.

[0061] During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed.

[0062] The calculation results of each computing node are collected to obtain the calculation results of the iterative task.

[0063] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0064] Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue;

[0065] The communication module is assigned to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative task according to the iterative execution queue.

[0066] During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed.

[0067] The calculation results of each computing node are collected to obtain the calculation results of the iterative task.

[0068] The aforementioned method, apparatus, computer equipment, storage media, and computer program products for parallel training and inference adaptation optimization of large models in next-generation heterogeneous supercomputing systems acquire model context relationships and inter-layer tensor dependencies. Based on these relationships, a directed acyclic dependency graph is constructed, and operators are encapsulated within this graph to obtain an iterative execution queue. Communication modules are allocated to each computing node in the parallel computing process, and each node executes corresponding batch iterative tasks according to the iterative execution queue. During the execution of the iterative execution queue, each computing node starts its operator engine within its allocated communication module, splits and rearranges tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed. The computation results from each computing node are then collected to obtain the iterative task computation results. Throughout this process, the iterative data stream is generated using a dependency graph, and batch tasks and communication modules are partitioned, realizing a complete set of large model training and inference operator libraries for heterogeneous supercomputing clusters. Batch partitioning is used to distribute iterative batch tasks to various computing nodes for parallel computation, achieving efficient data processing. Attached Figure Description

[0069] Figure 1 This is an application environment diagram for a new generation of heterogeneous supercomputing large model parallel training and inference adaptation and optimization method in one embodiment;

[0070] Figure 2 This is a flowchart illustrating an implementation of a method for parallel training and inference adaptation and optimization of a next-generation heterogeneous supercomputing large model.

[0071] Figure 3 This is a schematic diagram of a sub-process of S200 in one embodiment;

[0072] Figure 4 This is a diagram illustrating a dependency graph and an iterative execution queue generation method.

[0073] Figure 5 This is a schematic diagram of the communication module division in one embodiment;

[0074] Figure 6 This is a schematic diagram of the intra-operator parallelism method;

[0075] Figure 7 A schematic diagram illustrating the generation method of the training iteration execution queue;

[0076] Figure 8 This is a schematic diagram illustrating the structure of the operator engine.

[0077] Figure 9 This is a block diagram of a structural block diagram of a next-generation heterogeneous supercomputing large model parallel training and inference adaptation and optimization device in one embodiment.

[0078] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0080] The parallel training and inference adaptation optimization method for next-generation heterogeneous supercomputing large models provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 sends an optimization request to server 104. Server 104 responds to the optimization request, obtains the model context relationships and inter-layer tensor dependencies, constructs a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, encapsulates operators according to the directed acyclic dependency graph, and obtains an iterative execution queue. Communication modules are allocated to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative tasks according to the iterative execution queue. During the execution of the iterative execution queue, each computing node starts an operator engine within its allocated communication module, splits and rearranges tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed. The computation results of each computing node are collected to obtain the iterative task computation results. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0081] In one embodiment, such as Figure 2 As shown, a method for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0082] S200: Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue.

[0083] First, a deep analysis of the target large model is conducted to identify the model's contextual relationships (such as the flow of input data and the transmission of intermediate results) and inter-layer tensor dependencies (i.e., which layer's output is the input of subsequent layers). Based on this analysis, a Directed Acyclic Graph (DAG) is constructed, where nodes represent the model's layers or operators, and directed edges represent data dependencies. The DAG clearly demonstrates the model's execution flow, providing a foundation for subsequent task scheduling. According to the DAG, each layer or computational step of the model is encapsulated as an operator. These operators will serve as the basic units of parallel computing and will be allocated to different computing nodes for execution. Simultaneously, based on the dependencies in the DAG, an iterative execution queue is generated to guide the execution order of the operators.

[0084] S400: Distributes communication modules to each computing node in parallel computing, and each computing node executes the corresponding batch iterative tasks according to the iterative execution queue.

[0085] Based on the hardware architecture and characteristics of the heterogeneous supercomputing nodes, communication modules are rationally allocated. These communication modules are responsible for transmitting data and synchronizing states between computing nodes, and are crucial for achieving parallel computing. Computational tasks are assigned to each computing node according to an iterative execution queue. Each node executes its assigned operators in a predetermined order and according to dependencies. To further improve parallel efficiency, a dynamic task scheduling strategy can be introduced to dynamically adjust the allocation and execution order of tasks based on the real-time load and resource usage of the computing nodes.

[0086] S600: During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor calculations within the computing node through the communication module until the data in the iterative execution queue is processed.

[0087] On each computing node, an appropriate operator engine is launched based on the assigned task and operator. The operator engine is responsible for executing specific computational tasks and managing data input and output. To accommodate the demands of parallel computing, the computing node segments the input tensor and rearranges the data for parallel processing across multiple computing units. Through a communication module, computing nodes can efficiently transfer data and synchronize states. Within each computing node, parallel computing resources (such as GPUs and FPGAs) are used for tensor computation, thereby accelerating the model training and inference process.

[0088] Specifically, during the execution of the iterative queue, the computing node starts the operator engine within the communication module, splits and rearranges the tensor data, and the communication module activates the four computing clusters within the node through the pre-defined sub-blocks to complete parallel tensor computations within the operators until the queue is empty, thus completing this batch iteration task. In essence, the operator engine is responsible for completing all tensor computations in the execution queue for each iteration.

[0089] S800: Collects the computation results of each computing node to obtain the computation results of the iterative task.

[0090] After all computing nodes have completed their assigned tasks, the communication module collects the computation results from each node and sends them to the master node or a designated storage location. The collected results are then integrated and processed to obtain the final iterative task computation result. This process may include data merging, format conversion, and result verification.

[0091] In one embodiment, such as Figure 3 As shown, S200 includes:

[0092] S210: Obtain a request for model training and inference tasks.

[0093] The server first receives a task request for model training or inference. This request typically contains basic information about the model, such as its structure and the format of the input data.

[0094] S220: Responding to model training and inference task requests, obtain model context information.

[0095] After receiving a task request, the server loads and parses the model based on the information in the request, obtaining the model's context information. This context information includes the configuration of each layer of the model, the dimensions and types of the input and output tensors, etc.

[0096] S230: Obtain the tensor dependencies between model layers. Based on the model context information and the tensor dependencies between model layers, obtain the computation operators used to describe the tensor computation mode of this operation.

[0097] Next, the server needs to analyze the tensor dependencies between the layers within the model. This typically means determining which layers' outputs are the inputs of which layers, and the data flow paths between these inputs and outputs. Based on the contextual information and tensor dependencies obtained earlier, the system can generate a series of computation operators. These operators describe how each tensor in the model is computed and processed.

[0098] S240: For each tensor, a tensor description is used as a graph node. Each graph node encapsulates the tensor data type, computation operators, a pointer to the gradient tensor descriptor, and a pointer to the source tensor descriptor.

[0099] To better understand and optimize the model's execution flow, the system describes each tensor as a graph node. These nodes not only contain the tensor's data type but also encapsulate related computation operators, gradient tensor descriptors (for backpropagation), and source tensor descriptors (for tracing the data's origin).

[0100] S250: Extract all source tensor identifiers pointed to by the tensor identifiers in the final layer of the model to obtain the target source tensor identifier.

[0101] The server analyzes the tensors of the final layer of the model and identifies all related source tensors. These source tensors are necessary for computing the final layer tensors.

[0102] S260: Based on the target source tensor identifier, reverse connect all tensor corresponding operators of the model to generate a directed acyclic deep dependency graph.

[0103] With the information from the source tensor, the system can reverse-connect all tensors and their corresponding operators in the model, generating a directed acyclic deep dependency graph. This graph describes the dependencies between tensors and operators in the model, as well as the paths of data flow.

[0104] S270: Using the final layer tensor of the model as the initial node, a depth-first traversal is used to obtain the iterative execution queue.

[0105] Finally, the server uses the tensors of the model's final layer as initial nodes and traverses the entire dependency graph using a depth-first search approach. During this process, the system generates an iterative execution queue based on the order of data flow and dependencies. This queue describes the execution order of various computational operators and tensors during model execution.

[0106] In practical applications, for network training or inference tasks, model context information and tensor dependencies are obtained. Based on this context information, which includes the connection directions of each layer, the tensor computation mode of each layer, and the input model weights (the model weights are loaded into the model's preset initial tensors), and the tensor dependencies, computation operators describing the current tensor computation mode are obtained. For each tensor, a tensor descriptor is used as a graph node to encapsulate the tensor data and data type, computation operator, current tensor gradient tensor descriptor, and the source tensor descriptor that the current tensor points to. For example... Figure 4 As shown, each tensor is connected to its source tensor descriptor, and the tensor encapsulates the computation operators that the current tensor needs to perform, representing the operators that need to be called. Based on the source tensor identifier pointed to by the tensor identifier of the final layer of the model, all tensors of the model are connected in reverse to generate a directed acyclic depth dependency graph. The execution queue of the current iteration is obtained by depth-first traversal with the final layer tensor as the initial node, which is used as the execution queue of each iteration of the network.

[0107] In one embodiment, allocating a communication module to each computing node in parallel computing includes: obtaining the preset number of batch tasks corresponding to model training and inference, and the total number of computing nodes N that are enabled; allocating a communication block to each computing node according to the preset number of batch tasks and the total number of computing nodes N, wherein the communication module includes a single main block and multiple auxiliary blocks.

[0108] Specifically, the model needs to iterate over the entire execution queue as a whole, based on specific training and inference tasks. The number of iterations can be manually set by the user. For all N enabled computing nodes, the total number of batch tasks (baths) is divided into N parts, evenly distributed across the N computing nodes, and a communication module is assigned to each computing node. For example... Figure 5As shown, each communication module communicates with the other through the World_Comm communication domain based on the main block, and internally through the Wub_Somm communication domain. The communication module needs to adapt to each computing node and is divided into four sub-blocks. Each sub-block controls one acceleration cluster, thus the computations between the acceleration clusters corresponding to each sub-block are independent. Specifically, the acceleration cluster is a unique architecture of heterogeneous supercomputing, a specific computing region on a single processor of the supercomputing machine, as mentioned in the background technology. The communication module is our innovation. In a heterogeneous supercomputing cluster, there are multiple computing nodes, each with four acceleration clusters. Each node is controlled by a communication module, and the communication module consists of four sub-blocks, each controlling one of the four acceleration clusters on a computing node. Furthermore, these four sub-blocks are divided into one main block and three auxiliary blocks. The main block is responsible for data transmission and execution of the iteration queue, while the auxiliary blocks assist the main block in completing the corresponding parallel computations. Specifically, a communication module performs tensor parallel computations internally. The execution queue encapsulates a series of tensor descriptors, and the main block executes all tensor computations in the queue sequentially. Figure 6 As shown, in a specific tensor calculation, the main block sends tensor data and corresponding offsets to other auxiliary blocks. The main block and auxiliary blocks jointly complete the tensor calculation. After the calculation is completed, the main block collects the calculation results from the auxiliary blocks.

[0109] In one embodiment, for the training task, before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the method further includes:

[0110] Obtain the model training text corresponding to the training task; based on the model training file, determine the number of training sampling batches and the number of iterations for each iteration.

[0111] Training tasks typically consist of one or more files containing training instructions and data. Once the training files are obtained, they need to be parsed to determine the training parameters. Specifically, for a training task, before executing S100, the number of training sampling batches (batches) and the iteration count (iter) for each iteration need to be set. The batches of samples sampled in each iteration, based on the training text, will be used to train the model parameters on the number of batches of tokens within one iteration.

[0112] Obtain the model context relationships and inter-layer tensor dependencies. Construct a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies. Encapsulate operators based on the directed acyclic dependency graph to obtain the iterative execution queue, which includes:

[0113] Obtain the model context relationships and inter-layer tensor dependencies of the training task; construct a forward dependency graph based on the model context relationships and inter-layer tensor dependencies; construct a backward dependency graph based on the tensor dependencies calculated by gradients; concatenate the forward dependency graph and the backward dependency graph to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

[0114] Based on the model context relationships and inter-layer tensor dependencies of the training task, a forward dependency graph is constructed. Based on the tensor dependencies calculated by gradients, a backward dependency graph is constructed. The forward and backward dependency graphs are then concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies. For example... Figure 7 As shown, gradient calculation is required in the training task. After completing the forward propagation execution queue, the tensor descriptors are traversed sequentially. In addition to the calculation operators and source tensor descriptors required for the current tensor calculation, there is a gradient tensor descriptor used to determine whether the tensor needs gradient calculation. If the gradient tensor descriptor is not empty, it is used to encapsulate the gradient data index of the current tensor and set the source tensor descriptor of the current tensor's gradient as the gradient tensor descriptor of the current tensor. The operators in the gradient tensor descriptor are mainly row-wise multiplication and addition of tensors. Starting from the cross_entropy_back operator, the tensor gradients that need to be calculated during backward propagation are connected in the forward direction to complete the construction of the backward dependency graph.

[0115] In one embodiment, for a training task, a communication module is allocated to each computing node in parallel computing, and each computing node executes the corresponding batch iterative task according to the iterative execution queue, including:

[0116] The communication modules are assigned to each computing node in the parallel computing process, and the batch data of the training task is obtained. The batch data is evenly distributed to the corresponding communication modules of each computing node for computation. Each computing node performs the calculation of the forward propagation tensor and the backward propagation gradient tensor according to the iterative execution queue.

[0117] The batch data is evenly distributed to the corresponding communication modules of the computing nodes for computation. The forward propagation tensor and the backpropagation gradient tensor are processed by the operator engine according to the iterative execution queue. Specifically, the batch data batches are divided into batches / N data samples, and the batch data sample tensors are passed to the corresponding communication modules to start the operator engine for computation according to the iterative execution queue. Each computation is relatively independent. Each computing node averages the training gradient g of each sample g / batches and stores it in the current communication module.

[0118] The calculation results of each computing node are collected to obtain the calculation results of the iterative task. This includes: collecting the calculation results of the communication modules in each computing node, performing an all-reduce operation on the calculation results to synchronize global gradient data, and obtaining the calculation results of the iterative task.

[0119] After each communication module completes the batch training task required for the current iteration, an all-reduce operation is performed to synchronize global gradient data, and the master node updates the parameters based on the gradient data. After a certain number of iterations, the master node saves the currently trained model to local model data and begins the next training iteration, until the training task reaches the specified number of iterations or the model loss function is less than a preset value.

[0120] In one embodiment, for the inference task, before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the process further includes:

[0121] Receive the prompt statement required for model inference; set the number of batches of text sequence to be generated for inference; generate the text sequence based on the prompt statement and the number of batches of text sequence to be generated for inference.

[0122] The model receives prompts required for current inference and sets the number of batches (batches) to be generated from the text sequence, as well as the maximum length of the generated text sequence. The master node receives one or more prompts. When the input prompt is a single string, the model generates batches of text sequences based on that string. When the prompt consists of multiple strings, but not exceeding batches, the last prompt string will generate multiple text sequences. The number of input prompt strings must be less than the number of batches to be generated.

[0123] The process of allocating communication modules to each computing node in parallel computing, and having each computing node execute corresponding batch iterative tasks according to the iterative execution queue, includes: allocating communication modules to each computing node in parallel computing, with the communication module determining the number of iterations required based on the length of the text sequence to be inferred; and, based on the required number of iterations, having the communication module perform the corresponding batch iterative tasks in the iterative execution queue.

[0124] The batch inference task is evenly distributed among the communication modules. Each communication module determines the number of iterations required based on the length of the sequence to be generated. Each iteration executes the forward propagation queue to generate the token for that iteration. In each of the N communication modules, each module needs to complete the token generation task for batches / N sequences in each iteration. The token generation for each sequence is independent; different sequences can be generated sequentially within the same iteration. Each token generation requires a complete forward propagation computation. To improve inference generation efficiency, each communication module determines the length of the key-value cache (KV_Cache) in shared memory based on the current module's prompt length, the generated sequence length, and the model's hidden layer dimension. All key-value matrices calculated during the inference generation process are stored in the key-value cache of each communication module. In each attention calculation, the corresponding key-value matrix is ​​retrieved from the key-value cache to complete the forward computation.

[0125] In one embodiment, for the inference task, the computation results of each computing node are collected to obtain the iterative task computation results, including:

[0126] The computation results of each computing node are collected, and the text sequences generated by each computing node are extracted from the computation results of each computing node; the time when each computing node completes the inference generation task and the text sequence generated by each node are recorded.

[0127] After each communication module completes all batch iteration tasks, each module saves its generated text sequence. The master node then receives the time when each node completed its corresponding inference generation task and the text sequence generated by each communication module. The master node then concatenates the inference generation sequence of each communication module with the corresponding prompt statement as output and saves it to the appropriate text document.

[0128] In one embodiment, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit, and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0129] The operator engine is an abstraction of a communication module when it sequentially executes tensor computation tasks in the iterative execution queue. Based on the tensor data, tensor data type, tensor dimensions, offsets of each dimension of the tensor, the source tensor pointed to by the tensor, and its computation operators encapsulated within the iterative execution queue, it selects the corresponding computation mode, including but not limited to matrix pointwise multiplication, vector pointwise multiplication, matrix multiplication, normalization, softmax, and cross-entropy loss function. This part is mainly composed of four units: synchronization unit, device management unit, storage management unit, and core function unit.

[0130] In one embodiment, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block by the main component through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

[0131] Synchronization Unit: In the communication module, it is an abstraction of a heterogeneous computing node. Each communication module consists of four sub-blocks: a main block and three auxiliary blocks. This unit is responsible for logically dividing the input tensor and the output tensor. The main block distributes the tensor to each auxiliary block through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

[0132] Specifically, such as Figure 8 As shown, in the data synchronization unit of the current communication module, Process0 represents the main block of the communication module. The main block is responsible for maintaining and starting the sub-communication domains within the communication module. After receiving a specific tensor calculation task, the main block divides the tensor offset and tensor blocks for each auxiliary block according to the tensor data type and tensor data, and sends the corresponding tensor blocks to each auxiliary block. After receiving the tensor data sent by the main block, the auxiliary block, together with the main block and all other auxiliary blocks, completes the calculation of their respective tensor blocks. At the end of the calculation, the MPI communication method blocks the process, waiting for all sub-block tensor blocks to complete their calculations. Then, the main block is responsible for collecting the calculation results of each auxiliary block and using the result tensor data as the source tensor or queue calculation result for subsequent tensor calculations in the execution queue, either to perform the next tensor calculation or to declare the completion of the current iteration's execution queue calculation.

[0133] Device Management Unit: In a heterogeneous processor, this unit is responsible for controlling the maintenance of device instructions. The Host side represents the abstraction of the CPU instruction stream, and the Device side represents the abstraction of the DSP acceleration cluster. The heterogeneous supercomputing processor has four DSP clusters and corresponding cluster numbers. The startup, computation, and shutdown of the four Device sides are controlled by the main and auxiliary blocks of the communication module.

[0134] Specifically, each sub-block within the communication module corresponds to a Host. Each Device is independent of the others, but cannot run independently without the Host. The Host needs to ignite the Device based on the specific acceleration region number maintained by the current sub-block, create a Device thread group, and load the .dat file encapsulated in the core function to start the Device. At the same time, the Host is responsible for synchronizing with the Device. The Host submits task-related information to the created Device thread group. Based on the multi-core, multi-threaded characteristics of its internal acceleration region, each thread of the Device is independent of the others. The threads within the Device asynchronously complete the computation tasks submitted by the Host. The Host needs to wait for all the threads started by the Device to complete their assigned computation tasks before it can release the multi-threads of the Device and close the corresponding acceleration region.

[0135] Storage Management Unit: In heterogeneous processors, the Host and Device share the main memory; however, the Device can only access the storage space allocated to it by the Host. This unit is responsible for partitioning and rearranging tensor data according to the Device's multi-level storage method, and maintaining a one-dimensional pointer array to pass tensor data and their offsets in the linear storage mode to the Device.

[0136] Specifically, based on the various information contained in the tensor descriptor, including tensor data types such as F16, F32, and F64, tensor data including tensor data stored in a three-dimensional manner, tensor dimension dimensions including the hidden layer dimension, the token used in this calculation, the number of heads, and tensor dimension offsets including the number of bytes for the tensor offset in each dimension; since there are tensor transpose and permutation operators during the calculation process, these operators are logically implemented by only changing the tensor dimension offsets, which is inconsistent with the data transmission logic inside the communication module and is not coupled to the data retrieval method within the acceleration region, requiring a higher-level... The host, with its high-bandwidth memory, modifies the current tensor data arrangement. Secondly, since the device cannot directly access shared memory (DDR), the host needs to allocate corresponding shared memory space to the device through a specific interface and control its read / write permissions. The host saves the arranged tensors to the shared memory space allocated to the device, encapsulates all tensor data at their corresponding tensor offsets into a one-dimensional array, loads it into the device's thread group, and completes the tensor calculation. After the calculation is complete, the host releases the shared memory space with the device.

[0137] Core Function Unit: In heterogeneous processors, device-side computation and storage need to be accessed through specific interfaces. The device management unit starts a specific device, and the device receives a one-dimensional pointer array from the storage management unit to complete the multi-core, multi-threaded data partitioning within the DSP. By utilizing the device's unique vectorized instructions and DMA memory access optimizations and computational overlap, tensor parallel computation is completed.

[0138] Specifically, the execution of computation and memory access by the Device-side thread group requires the kernel function, which is encapsulated in the .dat file, to further segment the tensor data based on the corresponding offset and the total number of threads in the Device-side thread group after receiving the tensor data and its related data types, offsets, and computation modes. Each sub-block under the communication module maintains a Device-side thread group with 24 available cores. Each core corresponds to one computation thread. After allocating tensors according to the number of threads, the Device side allocates two cache spaces in the vector register AM (760KB) according to its length through a specific interface. The source tensor data is loaded from the shared memory space DDR line by line into the two AM blocks using the DMA method. It should be noted that the amount of tensor data loaded each time must be less than or equal to 380KB. The part of a line in the tensor that exceeds 380KB will be loaded into the scalar register SM (59KB). The tensors loaded into AM will be subjected to 16-word double-precision floating-point SIMD vectorization operations through the vector processing unit VPE unique to each Device-side core. The data that has been calculated is also written back to the shared memory space DDR using the DMA method. The loading and writing back processes can overlap with the calculation process. After the thread on the Device side completes the calculation, it will wait for other Device-side threads to complete their calculations in the synchronous blocking on the Host side.

[0139] In summary, the execution flow and operator engine proposed by the optimization method for large model parallel training and inference adapted to next-generation heterogeneous supercomputing can realize the deployment of large model training and inference tasks on heterogeneous supercomputing clusters. Furthermore, it couples large model operators with heterogeneous supercomputing processor architecture, and efficiently completes batch large model training and inference tasks by combining data parallelism and operator tensor parallelism.

[0140] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0141] Based on the same inventive concept, this application also provides a device for implementing the above-mentioned method for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the device for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models provided below can be found in the limitations of the method for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models described above, and will not be repeated here.

[0142] In one embodiment, such as Figure 9 As shown, a device for parallel training and inference adaptation and optimization of large models in next-generation heterogeneous supercomputing is provided, including:

[0143] The queue generation module 200 is used to obtain the model context relationship and the inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue.

[0144] The task allocation module 400 is used to allocate communication modules to each computing node in parallel computing, and each computing node executes the corresponding batch iterative tasks according to the iterative execution queue.

[0145] The parallel iteration module 600 is used to start the operator engine in the allocated communication module during the execution of the iteration execution queue, split and rearrange the tensor data, and perform parallel tensor calculations within the computing node through the communication module until the data in the iteration execution queue is processed.

[0146] The data collection module 800 is used to collect the calculation results of each computing node to obtain the calculation results of the iterative task.

[0147] In one embodiment, the queue generation module 200 is further configured to: acquire model training and inference task requests; respond to model training and inference task requests and acquire model context information; acquire tensor dependencies between model layers; and, based on the model context information and the tensor dependencies between model layers, obtain computation operators to describe the tensor computation mode; use tensor descriptions as graph nodes for each tensor, with each graph node encapsulating tensor data type, computation operator, a pointed-to gradient tensor descriptor, and a pointed-to source tensor descriptor; extract all pointed-to source tensor identifiers from the tensor identifiers of the final layer of the model to obtain the target source tensor identifier; reverse-connect all tensor operators corresponding to the model based on the target source tensor identifier to generate a directed acyclic depth dependency graph; and use depth-first traversal with the final layer tensor of the model as the initial node to obtain the iterative execution queue.

[0148] In one embodiment, the task allocation module 400 is further configured to obtain the preset batch number of tasks corresponding to model training and inference, and the total number of computing nodes N enabled; and to allocate a communication block to each computing node according to the preset batch number of tasks and the total number of computing nodes N. The communication module includes a single main block and multiple auxiliary blocks.

[0149] In one embodiment, the queue generation module 200 is further configured to: obtain the model training text corresponding to the training task; determine the number of training sampling batches and the number of iterations for each iteration based on the model training file; obtain the model context relationship and inter-layer tensor dependency relationship of the training task; construct a forward dependency graph based on the model context relationship and inter-layer tensor dependency relationship; construct a reverse dependency graph based on the tensor dependency relationship calculated by gradient; and concatenate the forward dependency graph and the reverse dependency graph to generate an iterative execution queue with tensor dependency relationship and related gradient tensor dependency.

[0150] In one embodiment, the task allocation module 400 is further configured to allocate communication modules to each computing node in parallel computing and obtain batch data of training tasks; distribute the batch data equally to the communication modules corresponding to each computing node for calculation; and have each computing node calculate the forward propagation tensor and the backward propagation gradient tensor according to the iterative execution queue.

[0151] The data collection module 800 is also used to collect the calculation results of the communication modules in each computing node, perform an all-reduce operation on the calculation results to synchronize global gradient data, and obtain the calculation results of the iterative task.

[0152] In one embodiment, the queue generation module 200 is further configured to receive a prompt statement required for model inference; set the number of batches of text sequences to be generated for inference; and generate a text sequence based on the prompt statement and the number of batches of text sequences to be generated for inference.

[0153] The task allocation module 400 is also used to allocate the communication module to each computing node in the parallel computing. The communication module determines the number of iterations required based on the length of the text sequence that needs to be inferred and generated. Based on the number of iterations required, the communication module performs the corresponding batch iterative tasks in the iterative execution queue.

[0154] In one embodiment, the data collection module 800 is also used to collect the calculation results of each computing node and extract the text sequence generated by each computing node from the calculation results of each computing node; and record the time when each computing node completes the inference generation task and the text sequence generated by each computing node.

[0155] In one embodiment, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit, and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0156] In one embodiment, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block by the main component through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

[0157] The modules in the aforementioned next-generation heterogeneous supercomputing large-scale model parallel training and inference adaptation optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0158] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores preset data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a parallel training and inference adaptation optimization method for next-generation heterogeneous supercomputing large models.

[0159] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method for parallel training and inference adaptation optimization of next-generation heterogeneous supercomputing large models.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for parallel training and inference adaptation optimization of large models for next-generation heterogeneous supercomputing.

[0162] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method for parallel training and inference adaptation optimization of large models for next-generation heterogeneous supercomputing.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for parallel training and inference adaptation and optimization of next-generation heterogeneous supercomputing large models, characterized in that, The method includes: Obtain the model context relationship and inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain the iterative execution queue; The communication module is assigned to each computing node in the parallel computing process, and each computing node executes the corresponding batch iterative task according to the iterative execution queue. During the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor computation within the computing node through the communication module until the data in the iterative execution queue is processed. The calculation results of each computing node are collected to obtain the calculation results of the iterative task; The process of obtaining model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on these relationships, and encapsulating operators within the graph to obtain an iterative execution queue includes: obtaining model training and inference task requests; responding to these requests by obtaining model context information; obtaining inter-layer tensor dependencies and, based on the context information and dependencies, obtaining computation operators to describe the current tensor computation pattern; using tensor descriptions as graph nodes for each tensor, with each node encapsulating a tensor data type, computation operator, a pointed-to gradient tensor descriptor, and a pointed-to source tensor descriptor; extracting all pointed-to source tensor identifiers from the final layer tensor identifiers to obtain target source tensor identifiers; reversing the connection of all tensor operators in the model based on the target source tensor identifiers to generate a directed acyclic depth dependency graph; and using a depth-first traversal with the final layer tensor as the initial node to obtain the iterative execution queue.

2. The method according to claim 1, characterized in that, The allocation of communication modules to each computing node in parallel computing includes: Obtain the preset number of batch tasks corresponding to model training and inference, and the total number of computing nodes N activated; Based on the preset number of batch tasks and the total number of computing nodes N, a communication block is allocated to each computing node. The communication module includes a single main block and multiple auxiliary blocks.

3. The method according to claim 1, characterized in that, Before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the process further includes: Obtain the model training text corresponding to the training task; Based on the model training file, determine the number of training sampling batches and the number of iterations for each iteration; The steps of obtaining model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on these relationships, and encapsulating operators according to the graph to obtain an iterative execution queue include: Obtain the model context relationships and inter-layer tensor dependencies for the training task; Based on the model context relationships and inter-layer tensor dependencies, a forward dependency graph is constructed; Based on the tensor dependency relationship calculated by gradient, construct the inverse dependency graph; The forward dependency graph and the reverse dependency graph are concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

4. The method according to claim 3, characterized in that The allocation of communication modules to each computing node in parallel computing, and the execution of corresponding batch iterative tasks by each computing node according to the iterative execution queue, includes: The communication module is assigned to each computing node in the parallel computing process, and the batch data of the training task is obtained. The batch data is evenly distributed to the communication modules corresponding to each computing node for computation. Each computing node performs the calculation of the forward propagation tensor and the backward propagation gradient tensor according to the iterative execution queue; The calculation results of each of the computing nodes are collected to obtain the calculation results of the iterative task, including: The calculation results of the communication modules in each computing node are collected, and an all-reduce operation is performed on the calculation results to synchronize global gradient data and obtain the calculation results of the iterative task.

5. The method according to claim 1, characterized in that, Before obtaining the model context relationships and inter-layer tensor dependencies, constructing a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulating operators according to the directed acyclic dependency graph to obtain the iterative execution queue, the process further includes: Receive prompts required for model inference; Set the number of batches of text sequences that need to be generated through inference; Based on the prompt statement and the number of batches to be generated according to the text sequence, generate the text sequence; The allocation of communication modules to each computing node in parallel computing, and the execution of corresponding batch iterative tasks by each computing node according to the iterative execution queue, includes: The communication module is assigned to each computing node in the parallel computing process, and the number of iterations required is determined by the communication module according to the length that the text sequence needs to be inferred and generated. Based on the required number of iterations, the communication module performs the corresponding batch iteration tasks in the iteration execution queue.

6. The method according to claim 5, characterized in that, The calculation results of each of the computing nodes are collected to obtain the calculation results of the iterative task, including: The calculation results of each computing node are collected, and the text sequences generated by each computing node are extracted from the calculation results of each computing node. Record the time it takes for each computing node to complete the inference generation task and the text sequence it generates.

7. The method according to claim 1, characterized in that, The computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit, and a core function unit; the communication module includes a single main block and three auxiliary blocks.

8. The method according to claim 7, characterized in that, The synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block by the main component through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

Images