Parallel training and reasoning adaptation optimization method for new-generation heterogeneous supercomputing large model

Abstract

The invention relates to a parallel training and reasoning adaptation optimization method for a new generation heterogeneous super-calculation large model, and the method comprises the steps: obtaining a model context relation and an interlayer tensor dependency relation, constructing a directed acyclic dependency relation graph based on the model context relation and the interlayer tensor dependency relation, packaging operators according to the directed acyclic dependency relation graph, and obtaining a new generation heterogeneous super-calculation large model. Obtaining an iterative execution queue; distributing the communication module to each computing node in parallel computing, and executing corresponding batch iteration tasks by each computing node according to the iteration execution queue; in the execution process of the iterative execution queue, each computing node starts an operator engine in the distributed communication module, segments and rearranges tensor data, and parallel tensor calculation in the computing nodes is performed through the communication module until data processing in the iterative execution queue is completed; and collecting the calculation result of each calculation node to obtain an iteration task calculation result. Iterative batch tasks are divided to each computing node for parallel computing by adopting a batch division means, so that efficient data processing is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for parallel training and reasoning adaptation optimization of a new generation of heterogeneous supercomputing large models. Background Art

[0002] In today's digital age, large model training and reasoning have become key technologies to promote the development of all walks of life in society. They are widely used in finance, medical care, education, autonomous driving, smart cockpit systems and other fields, and have greatly improved the level of intelligence in all walks of life by providing personalized services, optimizing decision-making processes and enhancing user experience.

[0003] At present, most terminal computing devices that have the computing power and storage required for large model training and reasoning, such as NVIDIA GPU, AMD GPU, Intel CPU, Google TPU, etc., need to have a high-performance computing library that supports the model data flow level and the underlying computing layer to support the large number of floating-point operations and huge memory overhead generated by large model training and reasoning.

[0004] The batch processing requirements for large model training and reasoning consume a lot of computing resources and memory, and a single machine often cannot meet these requirements. How to achieve efficient large model training and reasoning has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] Based on this, it is necessary to provide an efficient method, device, computer equipment, computer-readable storage medium and computer program product for parallel training and reasoning adaptation optimization of new-generation heterogeneous supercomputing large models in response to the above-mentioned technical problems.

[0006] In the first aspect, the present application provides a method for parallel training and reasoning adaptation optimization of large models for a new generation of heterogeneous supercomputers. The method comprises:

[0007] Obtaining a model context relationship and an inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0008] Allocating communication modules to each computing node in the parallel computing, so that each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0009] During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0010] The calculation results of each of the calculation nodes are collected to obtain the iterative task calculation result.

[0011] In one of the embodiments, the acquiring of the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue includes:

[0012] Obtain model training and inference task requests;

[0013] In response to the model training and reasoning task request, the model context information obtained;

[0014] Obtaining the tensor dependency relationship between model layers, and obtaining a calculation operator for describing the current tensor calculation mode according to the model context information and the tensor dependency relationship between model layers;

[0015] For each tensor, a tensor description is used as a graph node, each of the graph nodes encapsulates a tensor data type, a calculation operator, a gradient tensor descriptor pointed to, and a source tensor descriptor pointed to;

[0016] Extract all source tensor identifiers pointed to in the final layer tensor identifier of the model to obtain the target source tensor identifier;

[0017] Reversely connect all the operators corresponding to the model tensors according to the target source tensor identifier to generate a directed acyclic deep dependency graph;

[0018] The final layer tensor of the model is used as the initial node and a depth-first traversal is performed to obtain an iterative execution queue.

[0019] In one embodiment, allocating the communication module to each computing node in the parallel computing includes:

[0020] Get the preset batch task number corresponding to model training and reasoning, as well as the total number of enabled computing nodes N;

[0021] According to the preset number of batch tasks and the total number of computing nodes N, a communication block is allocated to each computing node, and the communication module includes a single main block and multiple auxiliary blocks.

[0022] In one embodiment, the method of obtaining the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue further includes:

[0023] Get the model training text corresponding to the training task;

[0024] Determine the number of training sample batches and the number of iterations for each iteration according to the model training file;

[0025] The acquiring of the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating the operator according to the directed acyclic dependency graph to obtain an iterative execution queue includes:

[0026] Obtaining the model context relationship and inter-layer tensor dependency relationship of the training task;

[0027] Constructing a forward dependency graph based on the model context relationship and the inter-layer tensor dependency relationship;

[0028] Build a reverse dependency graph based on tensor dependencies of gradient calculations;

[0029] The forward dependency graph and the reverse dependency graph are concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

[0030] In one embodiment, the assigning of the communication module to each computing node in the parallel computing, and each computing node executing the corresponding batch iteration task according to the iteration execution queue includes:

[0031] Allocate communication modules to each computing node in parallel computing and obtain batch data of the training task;

[0032] Distribute the batch data evenly to the communication modules corresponding to the computing nodes for computing;

[0033] Each of the computing nodes calculates a forward propagation tensor and a backward propagation gradient tensor according to the iterative execution queue;

[0034] The collecting the calculation results of each of the computing nodes to obtain the iterative task calculation result includes:

[0035] The calculation results of the communication modules in each of the computing nodes are aggregated, and an all-reduce operation is performed on the calculation results to synchronize global gradient data to obtain iterative task calculation results.

[0036] In one embodiment, the method of obtaining the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue further includes:

[0037] Receive the prompt statement required for model reasoning;

[0038] Set the number of batches that need to be generated for inference of text sequences;

[0039] Generate a text sequence according to the prompt sentence prompt and the batch number of the text sequence that needs to be inferred and generated;

[0040] The distributing of the communication module to each computing node in the parallel computing, and each computing node executing the corresponding batch iteration task according to the iteration execution queue includes:

[0041] Allocating a communication module to each computing node in the parallel computing, wherein the communication module determines the number of required iterations according to the length of the text sequence to be inferred and generated;

[0042] Based on the required number of iterations, the communication module iteratively executes the batch iteration tasks corresponding to the queue.

[0043] In one embodiment, the aggregating the calculation results of each of the computing nodes to obtain the iterative task calculation result includes:

[0044] Collecting the calculation results of each of the calculation nodes, and extracting the text sequences generated by each of the calculation results of the calculation nodes;

[0045] The time for each computing node to complete the reasoning generation task and the text sequence generated by each computing node are recorded.

[0046] In one of the embodiments, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0047] In one of the embodiments, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block through a scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through a gather method.

[0048] In the second aspect, the present application also provides a device for parallel training and reasoning adaptation optimization of large models for a new generation of heterogeneous supercomputing. The device includes:

[0049] A queue generation module is used to obtain the model context relationship and the inter-layer tensor dependency relationship, build a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate operators according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0050] A task allocation module is used to allocate the communication module to each computing node in the parallel computing, and each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0051] A parallel iteration module, used for each computing node to start an operator engine in the assigned communication module during the execution of the iterative execution queue, split and reorder tensor data, and perform parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0052] The data collection module is used to collect the calculation results of each of the computing nodes to obtain the iterative task calculation results.

[0053] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0054] Obtaining a model context relationship and an inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0055] Allocating communication modules to each computing node in the parallel computing, so that each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0056] During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0057] The calculation results of each of the calculation nodes are collected to obtain the iterative task calculation result.

[0058] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0059] Obtaining a model context relationship and an inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0060] Allocating communication modules to each computing node in the parallel computing, so that each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0061] During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0062] The calculation results of each of the calculation nodes are collected to obtain the iterative task calculation result.

[0063] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0064] Obtaining a model context relationship and an inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0065] Allocating communication modules to each computing node in the parallel computing, so that each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0066] During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0067] The calculation results of each of the calculation nodes are collected to obtain the iterative task calculation result.

[0068] The above-mentioned method, device, computer equipment, storage medium and computer program product for parallel training and reasoning adaptation optimization of large models for new generation heterogeneous supercomputing obtain the model context relationship and the inter-layer tensor dependency relationship, build a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate the operator according to the directed acyclic dependency graph to obtain an iterative execution queue; allocate communication modules to each computing node in the parallel computing, and each computing node executes the corresponding batch iterative task according to the iterative execution queue; during the execution of the iterative execution queue, each computing node starts the operator engine in the allocated communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed; the calculation results of each computing node are collected to obtain the iterative task calculation results. In the whole process, the dependency graph is used to generate the iterative data flow, and the batch tasks are divided with the communication module, so as to realize a complete set of large model training and reasoning operator libraries for heterogeneous supercomputing clusters, and the iterative batch tasks are divided into each computing node for parallel computing by batch partitioning, so as to realize efficient data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is an application environment diagram of a method for parallel training and reasoning adaptation optimization of a new generation of heterogeneous supercomputing large models in an embodiment;

[0070] Figure 2 It is a flowchart of a method for parallel training and reasoning adaptation optimization of a new generation of heterogeneous supercomputing large models in one embodiment;

[0071] Figure 3 is a schematic diagram of a sub-process of S200 in one embodiment;

[0072] Figure 4 A dependency graph and an iterative execution queue generation method graph;

[0073] Figure 5 A schematic diagram of the division of communication modules in one of the embodiments;

[0074] Figure 6 It is a schematic diagram of the intra-operator parallel mode;

[0075] Figure 7 A schematic diagram of how to generate a queue for training iteration execution;

[0076] Figure 8 This is a schematic diagram of the operator engine structure analysis;

[0077] Fig. 9 It is a structural block diagram of a device for parallel training and reasoning adaptation optimization of a new generation of heterogeneous supercomputing large models in an embodiment;

[0078] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] The parallel training and reasoning adaptation optimization method for the next-generation heterogeneous supercomputing large model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends an optimization request to the server 104, and the server 104 responds to the optimization request, obtains the model context relationship and the inter-layer tensor dependency relationship, builds a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulates the operator according to the directed acyclic dependency graph to obtain an iterative execution queue; assigns a communication module to each computing node in the parallel computing, and each computing node executes the corresponding batch iteration task according to the iterative execution queue; during the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed; the calculation results of each computing node are collected to obtain the calculation results of the iterative task. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0081] In one embodiment, Figure 2 As shown in the figure, a method for parallel training and reasoning optimization of large models for the next generation of heterogeneous supercomputing is provided. Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0082] S200: Obtain model context relationships and inter-layer tensor dependencies, construct a directed acyclic dependency graph based on the model context relationships and inter-layer tensor dependencies, and encapsulate operators according to the directed acyclic dependency graph to obtain an iterative execution queue.

[0083] First, conduct an in-depth analysis of the target large model to identify the contextual relationships of the model (such as the flow of input data, the transmission of intermediate results, etc.) and the tensor dependencies between layers (i.e., which layers’ outputs are the inputs of subsequent layers). Based on the above analysis, a directed acyclic graph (DAG) is constructed. The nodes in the graph represent the various layers or operators of the model, and the directed edges represent data dependencies. DAG can clearly show the execution process of the model and provide a basis for subsequent task scheduling. According to DAG, each layer or calculation step of the model is encapsulated as an operator. These operators will be assigned to different computing nodes as the basic units of parallel computing for execution. At the same time, based on the dependencies in the DAG, an iterative execution queue is generated to guide the execution order of the operators.

[0084] S400: Allocate the communication module to each computing node in the parallel computing, and each computing node executes the corresponding batch iteration task according to the iteration execution queue.

[0085] According to the hardware architecture of heterogeneous supercomputers and the characteristics of computing nodes, communication modules are reasonably allocated. These communication modules are responsible for transmitting data and synchronizing status between computing nodes, and are the key to achieving parallel computing. According to the iterative execution queue, computing tasks are assigned to each computing node. Each node executes its assigned operators in a predetermined order and dependency relationship. In order to further improve parallel efficiency, a dynamic task scheduling strategy can be introduced to dynamically adjust the allocation and execution order of tasks according to the real-time load and resource usage of computing nodes.

[0086] S600: During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and reorders the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed.

[0087] On each computing node, the corresponding operator engine is started according to the assigned tasks and operators. The operator engine is responsible for executing specific computing tasks and managing the input and output of data. In order to meet the needs of parallel computing, the computing node will split the input tensor and rearrange the data so that it can be processed in parallel on multiple computing units. Through the communication module, data and synchronization status can be efficiently transmitted between computing nodes. Inside each computing node, parallel computing resources (such as GPU, FPGA, etc.) are used to perform tensor calculations, thereby accelerating the training and reasoning process of the model.

[0088] Specifically, during the execution of the iteration queue, the computing node starts the operator engine in the communication module, splits and rearranges the tensor data, and the communication module enables the four computing clusters in the node through the divided sub-blocks to complete the parallel tensor calculation in the operator until the queue is empty, completing this batch iteration task. Specifically, the operator engine is responsible for completing all tensor calculations in the queue for each iteration execution.

[0089] S800: Collect the calculation results of each computing node to obtain the iterative task calculation result.

[0090] After all computing nodes complete their assigned tasks, the computing results of each node are collected to the master node or the designated storage location through the communication module. The collected results are integrated and processed to obtain the final iterative task computing results. This process may include data merging, format conversion, and result verification.

[0091] In one embodiment, if Figure 3 As shown, S200 includes:

[0092] S210: Obtain model training and reasoning task requests.

[0093] The server first receives a model training or inference task request. This request usually contains basic information about the model, such as the model structure and the format of the input data.

[0094] S220: Respond to model training and reasoning task requests and obtain model context information.

[0095] After receiving the task request, the server will load and parse the model according to the information in the request and obtain the context information of the model, including the configuration of each layer of the model, the dimensions and types of input and output tensors, etc.

[0096] S230: Obtain the tensor dependency between model layers, and obtain the calculation operator used to describe the current tensor calculation mode according to the model context information and the tensor dependency between model layers.

[0097] Next, the server needs to analyze the tensor dependencies between the layers within the model. This usually means determining which layers’ outputs are inputs to which layers, and the data flow paths between these inputs and outputs. Based on the contextual information and tensor dependencies obtained earlier, the system can generate a series of computational operators. These operators describe how each tensor in the model is calculated and processed.

[0098] S240: For each tensor, a tensor description is used as a graph node, and each graph node encapsulates a tensor data type, a calculation operator, a gradient tensor descriptor pointed to, and a source tensor descriptor pointed to.

[0099] In order to better understand and optimize the execution flow of the model, the system describes each tensor as a graph node. These nodes not only contain the data type of the tensor, but also encapsulate the related calculation operators, gradient tensor descriptors (for back propagation), and source tensor descriptors (for tracking the source of data).

[0100] S250: Extract all source tensor identifiers pointed to in the final layer tensor identifier of the model to obtain the target source tensor identifier.

[0101] The server analyzes the tensor of the final layer of the model and finds all the source tensors associated with it. These source tensors are required to calculate the final layer tensor.

[0102] S260: Reversely connect all the operators corresponding to the model tensors according to the target source tensor identifier to generate a directed acyclic deep dependency graph.

[0103] With the information of the source tensor, the system can reversely connect all tensors and corresponding operators in the model to generate a directed acyclic deep dependency graph. This graph describes the dependency between tensors and operators in the model, as well as the path of data flow.

[0104] S270: Using the model final layer tensor as the initial node, a depth-first traversal is performed to obtain an iterative execution queue.

[0105] Finally, the server will use the tensor of the final layer of the model as the initial node and traverse the entire dependency graph in a depth-first traversal manner. In this process, the system will generate an iterative execution queue according to the order of data flow and dependencies. This queue describes the execution order of each computing operator and tensor when the model is executed.

[0106] In actual application examples, for network training or inference tasks, model context information and tensor dependencies are obtained. According to the context information and tensor dependencies, the context information includes the connection direction of each layer of the model, the calculation mode of each layer of tensors and the input model weight. The model weight will be loaded into the preset initial tensor of the model; tensor dependencies, based on the context information, obtain the calculation operator used to describe the tensor calculation mode. For each tensor, the tensor descriptor is used as a graph node to encapsulate the tensor data and data type, calculation operator, current tensor gradient tensor descriptor, and the source tensor descriptor pointed to by the current tensor. Figure 4 As shown, each tensor is connected to its source tensor descriptor, and the calculation operator required for the current tensor is encapsulated in the tensor to represent the operator that needs to be called. Based on the source tensor identifier pointed to by the final layer tensor identifier of the model, all the tensors of the model are reversely connected to generate a directed acyclic deep dependency graph. The final layer tensor is used as the initial node and the depth-first traversal is used to obtain the execution queue of the current iteration as the execution queue of each iteration of the network.

[0107] In one embodiment, allocating a communication module to each computing node in a parallel computing includes: obtaining a preset number of batch tasks corresponding to model training and reasoning, and a total number of enabled computing nodes N; allocating a communication block to each computing node according to the preset number of batch tasks and the total number of computing nodes N, the communication module including a single main block and multiple auxiliary blocks.

[0108] Specifically, the model needs to use the entire model, that is, the entire execution queue, as the network iteration according to specific training and reasoning tasks. The number of network iterations can be manually set by the user. For all N enabled computing nodes, the total number of batch tasks, bathces, is divided into N parts and evenly distributed to the N computing nodes, and a communication module is assigned to each computing node. Figure 5As shown, each communication module communicates data through the World_Comm communication domain based on the main block, and each communication module communicates data through the Wub_Somm communication domain; the communication module needs to adapt to each computing node, and the communication module will be divided into four sub-blocks. Each sub-block will control an acceleration cluster, so the calculations between the acceleration clusters corresponding to each sub-block are independent of each other. Specifically, the acceleration cluster is a unique architecture of heterogeneous supercomputers, and is a specific computing area on a single processor of the supercomputer, which is mentioned in the background technology. The communication module is our innovation. There are multiple computing nodes in the heterogeneous supercomputer cluster, and each computing node has 4 acceleration clusters. Each node is controlled by a communication module. The communication module has 4 sub-blocks that control the 4 acceleration clusters on a computing node respectively. Secondly, these 4 sub-blocks will be divided into 1 main block and 3 auxiliary blocks. The main block is responsible for data transmission and the execution of the iteration queue, and the auxiliary block is responsible for assisting the main block to complete the corresponding parallel calculations. Specifically, a communication module has tensor parallel calculations, and a series of tensor descriptors are encapsulated in the execution queue. The main block will execute all tensor calculations in the queue in sequence. According to Figure 6 As shown, in a specific tensor calculation, the main block sends tensor data and corresponding offsets to other auxiliary blocks, and the main block and the auxiliary blocks jointly complete this tensor calculation. After the calculation is completed, the main block collects the calculation results of the auxiliary blocks.

[0109] In one embodiment, for a training task, a model context relationship and an inter-layer tensor dependency relationship are obtained, a directed acyclic dependency graph is constructed based on the model context relationship and the inter-layer tensor dependency relationship, and an operator is encapsulated according to the directed acyclic dependency graph, and before obtaining an iterative execution queue, the method further includes:

[0110] Obtain the model training text corresponding to the training task; determine the number of training sampling batches and the number of iterations for each iteration based on the model training file.

[0111] A training task usually consists of one or more files containing training instructions and data. Once the training files are obtained, they need to be parsed to determine the training parameters. Specifically, for a training task, before executing S100, it is necessary to set the number of training sample batches and the number of iterations iter for each iteration. Among them, the sample batches sampled in each iteration are based on the training text. In one iteration, the model parameters will train the tokens of the batches number.

[0112] Obtain the model context relationship and the inter-layer tensor dependency relationship, build a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate the operator according to the directed acyclic dependency graph to obtain the iterative execution queue including:

[0113] Obtain the model context relationship and inter-layer tensor dependency of the training task; construct a forward dependency graph based on the model context relationship and inter-layer tensor dependency; construct a reverse dependency graph based on the tensor dependency of gradient calculation; concatenate the forward dependency graph and the reverse dependency graph to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

[0114] Based on the model context relationship of the training task and the tensor dependency between layers, a forward dependency graph is constructed. Based on the tensor dependency of gradient calculation, a reverse dependency graph is constructed. The forward and reverse dependency graphs are concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies. Figure 7 As shown in the figure, the gradient calculation is required in the training task. After completing the forward propagation execution queue, the tensor descriptors are traversed in sequence. In addition to the calculation operators and source tensor descriptors required for this tensor calculation, there is a gradient tensor descriptor to determine whether the tensor needs to calculate the gradient. If the gradient tensor descriptor is not empty, it is used to encapsulate the gradient data index of the current tensor and set the source tensor descriptor of the source tensor gradient of the current tensor to the gradient tensor descriptor of the current tensor. The operators in the gradient tensor descriptor are mainly tensor multiplication and addition by row, starting from the cross_entropy_back operator, and the tensor gradients that need to be calculated for the forward connection back propagation are completed to complete the reverse dependency graph construction.

[0115] In one embodiment, for a training task, a communication module is assigned to each computing node in a parallel computing, and each computing node executes a corresponding batch iteration task according to an iteration execution queue, including:

[0116] Assign communication modules to each computing node in parallel computing and obtain batch data of training tasks; evenly distribute batch data to the communication modules corresponding to each computing node for calculation; each computing node calculates the forward propagation tensor and the back-propagation gradient tensor according to the iterative execution queue.

[0117] The batch data is evenly distributed to the communication module corresponding to the computing node for calculation. The forward propagation tensor and the backward propagation gradient tensor are completed by the operator engine according to the iterative execution queue. Among them, the batch data bathces are divided into batches / N data samples, and the batch data sample tensors are handed over to the corresponding communication module to start the operator engine for calculation according to the iterative execution queue. Each calculation is relatively independent. Each computing node averages the training gradient g of each sample g / batches and stores it in the current communication module.

[0118] Aggregating the calculation results of each computing node to obtain the iterative task calculation result includes: aggregating the calculation results of the communication modules in each computing node, performing an all-reduce operation on the calculation results to synchronize global gradient data, and obtaining the iterative task calculation result.

[0119] After each communication module completes the batch training task required for the current iteration, an all-reduce operation is performed to synchronize the global gradient data, and the master node completes the parameter update based on the gradient data. After a certain number of iterations, the master node saves the currently trained model to the local model data and performs the next training iteration task until the training task reaches the specified number of iterations or the model loss function is less than the preset value.

[0120] In one embodiment, for the reasoning task, the model context relationship and the inter-layer tensor dependency relationship are obtained, a directed acyclic dependency graph is constructed based on the model context relationship and the inter-layer tensor dependency relationship, and the operator is encapsulated according to the directed acyclic dependency graph. Before obtaining the iterative execution queue, the method further includes:

[0121] Receive the prompt statement prompt required for model reasoning; set the batch number of text sequences that need to be generated by reasoning; generate the text sequence according to the prompt statement prompt and the batch number of text sequences that need to be generated by reasoning.

[0122] The model receives the prompt statement required for the current model reasoning, and sets the batch number of text sequences that need to be inferred, as well as the maximum sequence length of the generated text. The prompt statement received by the master node can be one or more strings. When the input prompt statement is one string, the model will generate batches of text sequences based on the string prompt; when the prompt statement is multiple strings and is not greater than batches, the last string prompt will generate multiple text sequences; the number of input prompt statement strings must be less than the number of batches to be generated.

[0123] Allocating communication modules to each computing node in parallel computing, and having each computing node execute corresponding batch iteration tasks according to an iterative execution queue includes: allocating communication modules to each computing node in parallel computing, and having the communication module determine the required number of iterations according to the length of the text sequence to be inferred and generated; based on the required number of iterations, the communication module iteratively executes the corresponding batch iteration tasks of the queue.

[0124] The batch inference tasks are evenly divided into each communication module. The communication module will determine the number of iterations required according to the length of the sequence to be generated. Each iteration completes the inference generation of the token according to the forward propagation execution queue. Among them, in all N communication modules, each communication module needs to complete the token generation tasks required for inference of batches / N sequences in the current iteration in each round of iteration. There is no dependency relationship between the token inference generation of each sequence. In this iteration, the token inference generation of different sequences can be completed in sequence. The inference generation of each token needs to complete the complete forward propagation execution calculation in sequence. In order to improve the efficiency of inference generation, each communication module will determine the length of the key value cache KV_Cache in the shared memory according to the prompt length of the current module, the length of the generated sequence, and the dimension of the hidden layer of the model. The key value matrix calculated by all inference generation processes will be stored in the key value cache in each communication module. In each attention calculation, the corresponding key value matrix is ​​taken out from the key value cache to complete the forward calculation.

[0125] In one embodiment, for the reasoning task, the calculation results of each computing node are aggregated to obtain the iterative task calculation result, including:

[0126] Collect the calculation results of each computing node, and extract the text sequence generated by each computing node in the calculation results; record the time when each computing node completes the reasoning generation task and the text sequence generated by each computing node.

[0127] When each communication module completes all batch iteration tasks, each module saves its generated text sequence, and the master node receives the time when each node completes the corresponding inference generation task and the text sequence generated by each communication module. The master node concatenates the inference generation sequence of each communication module with the corresponding prompt sentence as output and saves it in the corresponding text document.

[0128] In one of the embodiments, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0129] The operator engine is an abstraction of a communication module when executing tensor computing tasks in the iterative execution queue in sequence. According to the tensor data encapsulated inside the iterative execution queue, the tensor data type, the tensor dimension, the offset of each dimension of the tensor, the source tensor pointed to by the tensor, and its computing operator, the corresponding computing mode is selected, including but not limited to matrix point-by-point multiplication, vector point-by-point multiplication, matrix multiplication, normalization, softmax and cross entropy loss function. This part is mainly composed of four units: synchronization unit, device management unit, storage management unit and core function unit.

[0130] In one of the embodiments, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block through a scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through a gather method.

[0131] Synchronization unit: In the communication module, it is an abstraction of a heterogeneous computing node. Each communication module consists of four sub-blocks: a main block and three auxiliary blocks. The unit is responsible for logically dividing the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

[0132] Specifically, Figure 8 As shown, in the data synchronization unit of the current communication module, Process0 represents the main block of the communication module, and the main block is responsible for maintaining and starting the sub-communication domain inside the communication module. After the main block receives the specific tensor calculation task, it divides the corresponding tensor offset and tensor block for each auxiliary block according to the tensor data type and tensor data, and sends the corresponding tensor block to each auxiliary block. After the auxiliary block receives the tensor data sent by the main block, the main block and all other auxiliary blocks jointly complete the calculation of their respective tensor blocks; at the end of the calculation, the MPI communication method is blocked, and after waiting for the completion of the calculation of all sub-block tensor blocks, the main block is responsible for collecting the calculation results of each auxiliary block, and uses the result tensor data as the source tensor or queue calculation result of the subsequent tensor calculation in the execution queue, and performs the next tensor calculation or declares that the execution queue calculation of this iteration is completed.

[0133] Device management unit: In heterogeneous processors, this unit is responsible for controlling the device instruction maintenance of heterogeneous processors, where the Host side represents the abstraction of the CPU instruction stream, and the Device side represents the abstraction of the DSP acceleration cluster. The heterogeneous supercomputer processor has four DSP clusters and corresponding cluster numbers, and controls the startup, calculation, and shutdown of the four Device sides through the main block and auxiliary block of the communication module.

[0134] Specifically, each sub-block inside the communication module corresponds to a Host end, and each Device end is independent of each other, but cannot run independently from the Host end. The Host end needs to ignite it according to the specific acceleration area number maintained by the current sub-block, create a Device end thread group, and load the .dat file encapsulated by the core function to start the Device end; at the same time, the Host end is responsible for synchronization with the Device end. The Host end will submit task-related information to the created Device end thread group. The Device end corresponds to the multi-core and multi-threaded characteristics of its internal acceleration area, and each thread is independent of each other. The threads inside the Device end asynchronously complete the computing tasks submitted by the Host end. The Host end needs to wait for the computing tasks assigned to all threads started by the Device end to be completed before releasing the multi-threads of the Device end and closing the corresponding acceleration area.

[0135] Storage management unit: In heterogeneous processors, the host and device share the main memory, but the device can only access the storage space allocated by the host. The host of this unit is responsible for dividing and rearranging tensor data according to the multi-level storage mode of the device, and maintaining a one-dimensional pointer array to pass tensor data and tensor offsets in linear storage mode to the device.

[0136] Specifically, according to the various types of information contained in the tensor descriptor, the tensor data types include F16, F32, and F64, the tensor data includes tensor data stored in three dimensions, the tensor dimension size includes the hidden layer dimension, the token and head number of this calculation, and the tensor dimension offset includes the number of bytes of each dimension tensor offset; since there are operators of tensor transposition and permutation types in the calculation process, this type of operator is implemented from a logical level by only changing the tensor dimension offset, which is inconsistent with the data transmission logic inside the communication module, and is not coupled to the data acquisition method inside the acceleration area, and needs to be implemented by a high The Host end that uses the fast memory bandwidth changes the current tensor data arrangement mode; secondly, the Device end cannot directly access the shared memory DDR, and the Host end needs to allocate the corresponding shared memory space to the Device end through a specific interface, and can control its read and write permissions. The Host end saves the tensors arranged above to the shared memory space allocated to the Device end, and encapsulates all tensor data into a one-dimensional array with the corresponding tensor offset, and loads it into the Device end thread group to complete the tensor calculation; after the calculation is completed, the Host end releases the memory space shared with the Device end.

[0137] Core function unit: In heterogeneous processors, the calculation and storage of the device side need to be accessed through a specific interface. The device management unit starts a specific device side, and the device side receives the one-dimensional pointer array passed by the storage management unit to complete the multi-core multi-threaded data partitioning inside the DSP, and uses the unique vectorized instructions of the device side and DMA memory access optimization and calculation overlap to complete tensor parallel computing.

[0138] Specifically, the execution of calculation and memory access of the device-side thread group needs to be performed through the kernel function encapsulated in the .dat file. After receiving the tensor data and its related data type, offset, and calculation mode, the tensor data needs to be further divided according to the corresponding offset and the total number of threads in the device-side thread group. The device-side thread group maintained by each sub-block under the communication module has 24 available cores, each core corresponds to a computing thread. After the tensors are allocated according to the number of threads, the device side allocates two cache spaces according to their length in the vector register AM (760KB) through a specific interface. The source tensor data is loaded from the shared memory space DDR into the two AM blocks two lines at a time using the DMA method. It should be noted that the amount of tensor data loaded each time needs to be less than or equal to 380KB. The part of a line in the tensor that exceeds 380KB will be loaded into the scalar register SM (59KB). The tensor loaded into AM will implement 16-word double-precision floating-point SIMD vectorized operations through the vector processing unit VPE unique to each device-side core. The data that has completed the calculation is also written back to the shared memory space DDR by the DMA method. The loading and writing back processes can overlap with the calculation process. After the thread on the device side completes the calculation, it will wait for other device-side threads to complete the calculation in the synchronous blocking on the host side.

[0139] In summary, through the execution process and operator engine proposed by the large-model parallel training and reasoning adaptation optimization method for the new generation of heterogeneous supercomputers, it is possible to deploy large model training and reasoning tasks in heterogeneous supercomputer clusters, and couple large model operators with heterogeneous supercomputer processor architectures. By combining data parallelism with tensor parallelism within operators, batch large model training and reasoning tasks can be completed efficiently.

[0140] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0141] Based on the same inventive concept, the embodiment of the present application also provides a device for parallel training and reasoning adaptation optimization of large models for new generation heterogeneous supercomputing, which is used to implement the above-mentioned method for parallel training and reasoning adaptation optimization of large models for new generation heterogeneous supercomputing. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the device for parallel training and reasoning adaptation optimization of large models for new generation heterogeneous supercomputing provided below can be referred to the limitations of the method for parallel training and reasoning adaptation optimization of large models for new generation heterogeneous supercomputing above, and will not be repeated here.

[0142] In one embodiment, Fig. 9 As shown, a device for parallel training and reasoning adaptation optimization of large models for a new generation of heterogeneous supercomputing is provided, including:

[0143] The queue generation module 200 is used to obtain the model context relationship and the inter-layer tensor dependency relationship, construct a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulate the operator according to the directed acyclic dependency graph to obtain an iterative execution queue;

[0144] The task allocation module 400 is used to allocate the communication module to each computing node in the parallel computing, and each computing node executes the corresponding batch iteration task according to the iteration execution queue;

[0145] The parallel iteration module 600 is used for each computing node to start the operator engine in the assigned communication module during the execution of the iterative execution queue, split and rearrange the tensor data, and perform parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed;

[0146] The data collection module 800 is used to collect the calculation results of each computing node to obtain the iterative task calculation results.

[0147] In one of the embodiments, the queue generation module 200 is also used to obtain model training and reasoning task requests; respond to model training and reasoning task requests, and obtain model context information; obtain tensor dependencies between model layers, and obtain a calculation operator used to describe the current tensor calculation mode based on the model context information and the tensor dependencies between model layers; use the tensor description as a graph node for each tensor, and each graph node encapsulates a tensor data type, a calculation operator, a pointed gradient tensor descriptor, and a pointed source tensor descriptor; extract all pointed source tensor identifiers in the model's final layer tensor identifier to obtain a target source tensor identifier; reversely connect all model tensor corresponding operators according to the target source tensor identifier to generate a directed acyclic deep dependency graph; use a depth-first traversal with the model's final layer tensor as the initial node to obtain an iterative execution queue.

[0148] In one of the embodiments, the task allocation module 400 is also used to obtain the preset number of batch tasks corresponding to model training and reasoning, and the total number of enabled computing nodes N; according to the preset number of batch tasks and the total number of computing nodes N, a communication block is allocated to each computing node, and the communication module includes a single main block and multiple auxiliary blocks.

[0149] In one of the embodiments, the queue generation module 200 is also used to obtain the model training text corresponding to the training task; determine the number of training sampling batches and the number of iterations for each iteration based on the model training file; obtain the model context relationship and inter-layer tensor dependency relationship of the training task; construct a forward dependency graph based on the model context relationship and the inter-layer tensor dependency relationship; construct a reverse dependency graph based on the tensor dependency relationship of gradient calculation; splice the forward dependency graph and the reverse dependency graph to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

[0150] In one embodiment, the task allocation module 400 is also used to allocate communication modules to each computing node in parallel computing, and obtain batch data of training tasks; evenly distribute the batch data to the communication modules corresponding to each computing node for calculation; and each computing node calculates the forward propagation tensor and the back propagation gradient tensor according to the iterative execution queue.

[0151] The data collection module 800 is also used to collect the calculation results of the communication modules in each computing node, perform an all-reduce operation on the calculation results to synchronize the global gradient data, and obtain the iterative task calculation results.

[0152] In one embodiment, the queue generation module 200 is further used to receive a prompt statement prompt required for model reasoning; set the batch number of text sequences that need to be reasoned and generated; generate a text sequence according to the prompt statement prompt and the batch number of text sequences that need to be reasoned and generated;

[0153] The task allocation module 400 is also used to allocate the communication module to each computing node in the parallel computing, and the communication module determines the required number of iterations according to the length of the text sequence that needs to be inferred and generated; based on the required number of iterations, the communication module iterates and executes the corresponding batch iteration tasks of the queue.

[0154] In one embodiment, the data collection module 800 is also used to collect the calculation results of each computing node, and extract the text sequence generated by each computing node in the calculation results; record the time when each computing node completes the reasoning generation task and the text sequence generated by each computing node.

[0155] In one of the embodiments, the computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit and a core function unit; and the communication module includes a single main block and three auxiliary blocks.

[0156] In one of the embodiments, the synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block through a scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through a gather method.

[0157] Each module in the above-mentioned device for parallel training and reasoning adaptation optimization of large models for new-generation heterogeneous supercomputing can be implemented in whole or in part through software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0158] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for parallel training and reasoning adaptation optimization of a new generation of heterogeneous supercomputing large models is implemented.

[0159] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the above-mentioned parallel training and reasoning adaptation optimization method for the new generation of heterogeneous supercomputing large models is implemented.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned parallel training and reasoning adaptation optimization method for the new generation of heterogeneous supercomputing large models is implemented.

[0162] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the above-mentioned parallel training and reasoning adaptation optimization method for the new generation of heterogeneous supercomputing large models.

[0163] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for parallel training and reasoning optimization of large models for new-generation heterogeneous supercomputing, characterized in that: The method comprises: Obtaining a model context relationship and an inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue; Allocating communication modules to each computing node in the parallel computing, so that each computing node executes the corresponding batch iteration task according to the iteration execution queue; During the execution of the iterative execution queue, each computing node starts the operator engine in the assigned communication module, splits and rearranges the tensor data, and performs parallel tensor calculations in the computing node through the communication module until the data in the iterative execution queue is processed; The calculation results of each of the calculation nodes are collected to obtain the iterative task calculation result.

2. The method according to claim 1, characterized in that The acquiring of the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating the operator according to the directed acyclic dependency graph to obtain an iterative execution queue includes: Obtain model training and inference task requests; In response to the model training and reasoning task request, the model context information obtained; Obtaining the tensor dependency relationship between model layers, and obtaining a calculation operator for describing the current tensor calculation mode according to the model context information and the tensor dependency relationship between model layers; For each tensor, a tensor description is used as a graph node, each of the graph nodes encapsulates a tensor data type, a calculation operator, a gradient tensor descriptor pointed to, and a source tensor descriptor pointed to; Extract all source tensor identifiers pointed to in the final layer tensor identifier of the model to obtain the target source tensor identifier; Reversely connect all the operators corresponding to the model tensors according to the target source tensor identifier to generate a directed acyclic deep dependency graph; The final layer tensor of the model is used as the initial node and a depth-first traversal is performed to obtain an iterative execution queue.

3. The method according to claim 1, characterized in that Assigning communication modules to each computing node in parallel computing includes: Get the preset batch task number corresponding to model training and reasoning, as well as the total number of enabled computing nodes N; According to the preset number of batch tasks and the total number of computing nodes N, a communication block is allocated to each computing node, and the communication module includes a single main block and multiple auxiliary blocks.

4. The method according to claim 1, characterized in that: The method further includes obtaining the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue: Get the model training text corresponding to the training task; Determine the number of training sample batches and the number of iterations for each iteration according to the model training file; The acquiring of the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating the operator according to the directed acyclic dependency graph to obtain an iterative execution queue includes: Obtaining the model context relationship and inter-layer tensor dependency relationship of the training task; Constructing a forward dependency graph based on the model context relationship and the inter-layer tensor dependency relationship; Build a reverse dependency graph based on tensor dependencies of gradient calculations; The forward dependency graph and the reverse dependency graph are concatenated to generate an iterative execution queue with tensor dependencies and related gradient tensor dependencies.

5. The method according to claim 4, characterized in that The distributing of the communication module to each computing node in the parallel computing, and each computing node executing the corresponding batch iteration task according to the iteration execution queue includes: Allocate communication modules to each computing node in parallel computing and obtain batch data of the training task; Distribute the batch data evenly to the communication modules corresponding to the computing nodes for computing; Each of the computing nodes calculates a forward propagation tensor and a backward propagation gradient tensor according to the iterative execution queue; The collecting the calculation results of each of the computing nodes to obtain the iterative task calculation results includes: The calculation results of the communication modules in each of the computing nodes are aggregated, and an all-reduce operation is performed on the calculation results to synchronize global gradient data to obtain iterative task calculation results.

6. The method according to claim 1, characterized in that The method further includes obtaining the model context relationship and the inter-layer tensor dependency relationship, constructing a directed acyclic dependency graph based on the model context relationship and the inter-layer tensor dependency relationship, and encapsulating operators according to the directed acyclic dependency graph to obtain an iterative execution queue: Receive the prompt statement required for model reasoning; Set the batch size of text sequences that need to be inferred and generated; Generate a text sequence according to the prompt sentence prompt and the batch number of the text sequence that needs to be inferred and generated; The distributing of the communication module to each computing node in the parallel computing, and each computing node executing the corresponding batch iteration task according to the iteration execution queue includes: Allocating a communication module to each computing node in the parallel computing, wherein the communication module determines the number of required iterations according to the length of the text sequence to be inferred and generated; Based on the required number of iterations, the communication module iteratively executes the batch iteration tasks corresponding to the queue.

7. The method according to claim 6, characterized in that The collecting the calculation results of each of the computing nodes to obtain the iterative task calculation result includes: Collecting the calculation results of each of the calculation nodes, and extracting the text sequences generated by each of the calculation results of the calculation nodes; The time for each computing node to complete the reasoning generation task and the text sequence generated by each computing node are recorded.

8. The method according to claim 1, characterized in that The computing node includes a heterogeneous supercomputing processor; the operator engine includes a data synchronization unit, a device management unit, a storage management unit and a core function unit; the communication module includes a single main block and three auxiliary blocks.

9. The method according to claim 8, characterized in that The synchronization unit is used to logically divide the input tensor and the output tensor, and the main block distributes the tensor to each auxiliary block through the scatter method. After the calculation is completed, the main block collects the calculation results of each auxiliary block through the gather method.

Images