A model training system and method, a storage medium and an electronic device

Abstract

The present disclosure relates to a model training system and method, a storage medium and an electronic device, the system comprising: a large language model training framework receiving a model instance and a compilation parameter input by a user; the large language model training framework transmitting the model instance and the compilation parameter into a port of a model static graph optimization compiler, and requesting to compile the model instance; the model static graph optimization compiler receiving the transmitted model instance and the compilation parameter, performing a graph compilation operation on the model instance by using a corresponding compilation backend of the compilation parameter, obtaining a static graph of the model instance, and returning the static graph to the large language model training framework; and the large language model training framework receiving the static graph, and training the model instance based on the static graph. The embodiments of the present disclosure can improve the training efficiency and resource utilization.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training system and method, storage medium and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are being used more and more widely in various fields. These models, based on deep learning technology, have achieved powerful natural language processing capabilities. However, the training process of large language models requires processing massive amounts of data and complex model parameters, placing extremely high demands on computing resources.

[0003] The Large Language Model Training Framework is a framework for training large language models. It aims to improve the efficiency and scalability of large-scale model training by using various techniques such as mixed precision and data parallelism.

[0004] However, despite significant improvements in training efficiency of large language model training frameworks, the processing resources and costs of training large models remain high as the model size increases. Summary of the Invention

[0005] This disclosure proposes a model training technique.

[0006] According to one aspect of this disclosure, a model training system is provided, comprising:

[0007] A large language model training framework that receives model instances and compilation parameters from the user;

[0008] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests the compilation of the model instance.

[0009] The model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operation on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework.

[0010] The large language model training framework receives the static graph and trains the model instance based on the static graph.

[0011] In one possible implementation, if the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform the compilation operation.

[0012] The first compiler obtains the bytecode of the model instance;

[0013] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies a module in the model instance that does not support compilation into a static graph, it skips the compilation operation for the module that does not support compilation into a static graph; when the first compiler identifies a module in the model instance that it does not support compilation, it replaces the module that does not support compilation with a functionally equivalent module and performs a compilation operation on the code of the equivalent module.

[0014] In one possible implementation, the module that does not support compilation as a static graph includes a recomputation module, and the first compiler skips the compilation operation of the recomputation module when it recognizes the recomputation module of the model instance.

[0015] The first compiler does not support modules that perform compilation operations, including in-place array operation modules. When the code of an in-place array operation module used for in-place array operations is identified, the code of the in-place array operation module is replaced with the code of a non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module.

[0016] In one possible implementation, the large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data.

[0017] The large language model training framework disables recompilation when the amount of data in the last batch of training data is less than the amount of data in the previous batches, and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

[0018] In one possible implementation, if the compilation backend indicated by the compilation parameters includes a second compiler for operator transformation, the model static graph optimizer calls the second compiler to perform the compilation operation;

[0019] The second compiler captures the computation graphs of forward and backward propagation of model instances and generates a joint computation graph;

[0020] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator is identified in the large language model training framework, the custom operator is encapsulated into the operation of the model static graph optimizer using the port of the model static graph optimizer.

[0021] In one possible implementation, if the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform the compilation operation.

[0022] The third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, thus obtaining a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped.

[0023] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0024] In one possible implementation, if the compilation backend indicated by the compilation parameters includes a fourth compiler for adaptation to a specific hardware platform, the model static graph optimizing compiler calls the fourth compiler to perform the compilation operation.

[0025] The fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform; wherein, when a set communication operator for integrating the processing results of multiple processors is identified, the function of the set communication operator in the backpropagation computation graph is converted into: retrieving the set communication results saved in the forward propagation stage from memory;

[0026] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0027] According to one aspect of this disclosure, a model training method is provided for application in the aforementioned system, comprising:

[0028] A large language model training framework that receives model instances and compilation parameters from the user;

[0029] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests the compilation of the model instance.

[0030] The model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operation on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework.

[0031] The large language model training framework receives the static graph and trains the model instance based on the static graph.

[0032] In one possible implementation, the graph compilation operation on the model instance includes:

[0033] When the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform the compilation operation;

[0034] The first compiler obtains the bytecode of the model instance;

[0035] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies a module in the model instance that does not support compilation into a static graph, it skips the compilation operation for the module that does not support compilation into a static graph; when the first compiler identifies a module in the model instance that it does not support compilation, it replaces the module that does not support compilation with a functionally equivalent module and performs a compilation operation on the code of the equivalent module.

[0036] In one possible implementation, the module that does not support compilation to a static graph includes a recomputation module, and the graph compilation operation on the model instance includes:

[0037] When the first compiler identifies a recomputation module of the model instance, it skips the compilation operation for the recomputation module.

[0038] In one possible implementation, the first compiler does not support modules that perform compilation operations, including in-place array operation modules. The graph compilation operation on the model instance includes:

[0039] When the code of the in-place array operation module used for in-place array operations is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and the code of the non-in-place array operation module is compiled.

[0040] In one possible implementation, training the model instance based on the static graph includes:

[0041] The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data.

[0042] The large language model training framework disables recompilation when the amount of data in the last batch of training data is less than the amount of data in the previous batches, and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

[0043] In one possible implementation, the graph compilation operation on the model instance includes:

[0044] When the compilation backend indicated by the compilation parameters includes a second compiler for operator transformation, the model static graph optimization compiler calls the second compiler to perform the compilation operation.

[0045] The second compiler captures the computation graphs of forward and backward propagation of model instances and generates a joint computation graph;

[0046] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator is identified in the large language model training framework, the custom operator is encapsulated into the operation of the model static graph optimizer using the port of the model static graph optimizer.

[0047] In one possible implementation, the graph compilation operation on the model instance includes:

[0048] When the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform the compilation operation;

[0049] The third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, thus obtaining a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped.

[0050] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0051] In one possible implementation, the graph compilation operation on the model instance includes:

[0052] When the compilation backend indicated by the compilation parameters includes a fourth compiler adapted for a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform the compilation operation;

[0053] The fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform; wherein, when a set communication operator for integrating the processing results of multiple processors is identified, the function of the set communication operator in the backpropagation computation graph is converted into: retrieving the set communication results saved in the forward propagation stage from memory;

[0054] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0055] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0056] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0057] In this embodiment, the large language model training framework receives a model instance and compilation parameters input by the user, and then passes the model instance and compilation parameters to the port of the model static graph optimization compiler to request compilation of the model instance. The model static graph optimization compiler receives the passed model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operations on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework. The large language model training framework receives the static graph and trains the model instance based on the static graph. Therefore, by introducing a static graph optimization compiler into the large language model training framework, the computational and memory requirements during training and inference are reduced because static graphs can eliminate redundant calculations and intermediate variable storage common in dynamic graphs. This allows for more efficient use of hardware resources, reduces unnecessary computational and memory overhead during training, and improves training efficiency and resource utilization.

[0058] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0060] Figure 1 A block diagram of a model training system according to an embodiment of the present disclosure is shown.

[0061] Figure 2 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown.

[0062] Figure 3 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0063] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0064] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0065] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0066] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0067] Figure 1 A block diagram of a model training system according to an embodiment of the present disclosure is shown, such as Figure 1 As shown, the system includes:

[0068] The large language model training framework 101 receives model instances and compilation parameters from the user.

[0069] The large language model training framework 101 passes the model instance and compilation parameters to the port of the model static graph optimization compiler 102, requesting the compilation of the model instance;

[0070] The model static graph optimizer 102 receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operation on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework.

[0071] The large language model training framework 101 receives the static graph and trains the model instance based on the static graph.

[0072] Large language model training frameworks can be software or systems used to train large natural language processing models. They provide the infrastructure and tools needed to train large-scale language models, enabling the construction and training of large language models.

[0073] Large language model training frameworks can include multiple components, such as model definition interfaces, data preprocessing tools, optimization algorithm libraries, and distributed training components. In one example, a large language model training framework could be used to train a large Transformer model. Because it supports various parallel strategies (such as data parallelism, model parallelism, and pipelined parallelism) and mixed-precision training, it can efficiently train large language models using multi-GPU or multi-node clusters.

[0074] A model instance can be a large language model to be trained. The model instance can define functional modules and parameters based on the large language model training framework to achieve specific functions. This model instance can be a neural network model. The model instance is the object operated on by the large language model training framework. Based on the input training data and training objective, the large language model can calculate prediction results through forward propagation and then update weight parameters through backpropagation, thus achieving the training of the large language model.

[0075] The large language model training framework can receive model instances and compilation parameters input by users. Users can provide model instances to be trained and compilation parameters to the large language model training framework through the user graphical interface or interface provided by the framework. This allows users to flexibly select models and compilation strategies to adapt to different training needs and hardware environments, thereby improving training efficiency and effectiveness.

[0076] The static graph optimizer compiler generates optimized model representations (i.e., static graphs) through static graph analysis and optimization techniques, thereby improving model execution efficiency and resource utilization. Based on user-inputted compilation parameters, the static graph optimizer compiler invokes the corresponding compiler to perform compilation operations, analyzes the structure and parameters of model instances, and generates static graphs.

[0077] Compilation parameters can be used to control the compilation behavior of the model. These parameters define the optimization strategies, compilation backends, and other possible settings used during compilation. The compilation backend may include: a first compiler for capturing the forward propagation computation graph; a second compiler for performing operator transformations; and a third compiler for performing deep-level optimizations of the computation graph. Specific implementations can be found in the possible implementations provided in this disclosure, and will not be elaborated upon here.

[0078] Large language model training frameworks pass user-provided model instances and compilation parameters to a static graph optimizer (such as Torch Compile), requesting the model to be compiled to generate an optimized static graph representation. The static graph optimizer generates the optimized static graph based on the model instance and compilation parameters. By introducing the static graph optimizer, the model can be optimized before training, improving computational efficiency and resource utilization during the large language model training process, thereby reducing computational redundancy and memory utilization.

[0079] The static graph optimizer compiler performs graph compilation operations based on the received model instances and compilation parameters, generating optimized static graph representations and returning them to the large language model training framework. During graph compilation, the compilation backend analyzes and transforms the model instances, analyzing their structure and attributes, and then optimizes the large language model according to optimization rules (such as operator fusion, memory reuse, and parallel execution) to generate more efficient and resource-saving static graph representations. The compilation backend then returns the optimized static graph to the large language model training framework for subsequent training operations.

[0080] After receiving a static graph, the large language model training framework trains model instances based on it. The framework parses the structure and parameters of the static graph and trains according to the computational flow and data flow defined within it. For example, upon receiving the optimized static graph of the model instances returned by Torch Compile, the framework trains according to the optimized computational flow and data flow defined in the static graph, significantly improving training speed and efficiency.

[0081] In this embodiment, the large language model training framework receives a model instance and compilation parameters input by the user, and then passes the model instance and compilation parameters to the port of the model static graph optimization compiler to request compilation of the model instance. The model static graph optimization compiler receives the passed model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operations on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework. The large language model training framework receives the static graph and trains the model instance based on the static graph. Therefore, by introducing a static graph optimization compiler into the large language model training framework, the computational and memory requirements during training and inference are reduced because static graphs can eliminate redundant calculations and intermediate variable storage common in dynamic graphs. This allows for more efficient use of hardware resources, reduces unnecessary computational and memory overhead during training, and improves training efficiency and resource utilization.

[0082] In one possible implementation, when the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimizer calls the first compiler to perform a compilation operation; the first compiler obtains the bytecode of the model instance; the first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies a module in the model instance that does not support compilation into a static graph, it skips the compilation operation for the module that does not support compilation into a static graph; when the first compiler identifies a module in the model instance that it does not support performing a compilation operation, it replaces the module that does not support performing a compilation operation with a functionally equivalent module and performs a compilation operation on the code of the equivalent module.

[0083] Users specify a compilation backend through compilation parameters. If the specified backend is the first compiler, the model static graph optimization compiler will call the first compiler to capture and process the model's forward propagation computation graph. The first compiler can process the forward propagation computation graph of model instances, obtaining the static graph of the model instances. For example, if a user enters the compilation parameter "aaa" through the command line, the model static graph optimization compiler will recognize that the compilation backend corresponding to this parameter is the first compiler, and will call the first compiler to capture and process the model's forward propagation computation graph, compiling the model instances to generate a static graph. This converts the original dynamic graph model into a static graph model.

[0084] During the compilation process, the first compiler obtains the bytecode of the model instance, that is, it extracts the bytecode required to perform the compilation operation from the model instance. Bytecode is the low-level representation of the model instance when it is executed in computer memory. The first compiler obtains the structure and calculation process of the model by parsing the bytecode.

[0085] Specifically, the compiler analyzes the operations and data flow in the bytecode, identifies the forward propagation path of the model, and transforms this information into a static graph structure through optimization and refactoring for subsequent training and inference.

[0086] Since there are modules in the model instance that do not support compilation into static graphs, during the compilation process, when a module that does not support compilation into static graphs is detected in the model instance, the compilation operation for the module that does not support compilation into static graphs can be skipped. In other words, the compilation operation for the module that does not support compilation into static graphs is not performed to avoid compilation errors. The module that does not support compilation into static graphs can still be executed in the form of dynamic graphs.

[0087] Furthermore, if a module in a model instance is identified that is not supported by the first compiler for compilation, in order to improve training efficiency and resource utilization without affecting the complete functionality of the model, the module that does not support compilation can be replaced with a functionally equivalent module, and the code of the equivalent module can be compiled. The equivalent module and the module that is not supported by the first compiler implement the same functionality, although the implementation process may differ, but this will not affect the training effect on large models.

[0088] After replacing modules that the first compiler does not support performing compilation operations with other functionally equivalent operations, the first compiler will compile the replaced operations to obtain a static graph of the model instances.

[0089] In this embodiment of the disclosure, when the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation; the first compiler obtains the bytecode of the model instance; the first compiler compiles the bytecode to construct the static graph of the model. Since the model instance is defined and input based on the large language model training framework, there may be situations where the model static graph optimization compiler is incompatible. Therefore, when a module in the model instance is identified that does not support compilation into a static graph, the compilation operation for the module that does not support compilation into a static graph can be skipped; and when a module in the model instance is identified that the first compiler does not support performing a compilation operation, the module that does not support performing a compilation operation is replaced with a functionally equivalent module, and the code of the equivalent module is compiled. Therefore, adaptation to the large language model training framework and the static graph optimization compiler is achieved. The adapted large language model training framework can train model instances in static graph mode. Through graph compilation and graph optimization techniques, it can make more effective use of GPU resources, reduce runtime overhead, and thus improve training efficiency. Compared with dynamic graph mode, the overall training efficiency is improved by about 15%.

[0090] In one possible implementation, the modules that do not support compilation into static graphs include recomputation modules. When the first compiler identifies a recomputation module for the model instance, it skips the compilation operation for the recomputation module. Modules that the first compiler does not support performing compilation operations include in-place array operation modules. When the code of an in-place array operation module for performing in-place array operations is identified, the code of the in-place array operation module is replaced with the code of a non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module.

[0091] In this implementation, the modules that do not support compilation into static graphs include recomputation modules. Therefore, when the first compiler identifies a recomputation module in a model instance, it cannot convert the recomputation module into a static graph, and directly compiling the recomputation module would result in an error. Thus, the first compiler can skip compiling the recomputation module. The recomputation module is used to determine which layers' activation values ​​in the model instance should be recomputed to achieve a balance between memory usage and computational efficiency. This requires the ability to dynamically construct and modify the computation graph, which is feasible in dynamic graph execution mode. Therefore, the recomputation module cannot be converted into a static graph. The specific execution process of the recomputation module is not described in detail in this disclosure.

[0092] For example, in related technologies, a recomputation mechanism is implemented by calling the PyTorch utility function `torch.utils.checkpoint` to reduce memory usage. However, when the first compiler attempts to compile a model containing these recomputation modules, an error occurs. Therefore, in this implementation, the compilation of the recomputation modules can be skipped.

[0093] In this implementation, the first compiler does not support modules that perform compilation operations, including in-place array operation modules. Therefore, when the first compiler recognizes an in-place array operation module, it will convert it into a non-in-place array operation module and then perform the compilation operation.

[0094] In-place array operations are operations that modify the original array or memory space directly. These operations are not supported by the first-line compiler because they can compromise data consistency. Therefore, when an in-place array operation module is detected, the first-line compiler converts it into an equivalent form that operates on the new memory space (i.e., a non-in-place array operation), thus avoiding compilation errors and ensuring data integrity.

[0095] For example, in related technologies, in-place array operations may exist. However, the first compiler does not support such in-place array operations, so it will convert them into non-in-place array operation forms, so that the calculation results are stored in a new memory space, thereby avoiding potential compilation errors and data consistency issues.

[0096] In this embodiment, the modules that do not support compilation into static graphs include recomputation modules. When the first compiler identifies a recomputation module for the model instance, it skips the compilation operation for that recomputation module. Modules that the first compiler does not support compiling include in-place array operation modules. When the code of an in-place array operation module is identified, the code of the in-place array operation module is replaced with the code of a non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module. Thus, by skipping the compilation of recomputation modules that do not support compilation into static graphs, compilation errors caused by the compiler's lack of support for that module are avoided. By converting in-place array operation modules to non-in-place array operations, it is ensured that even in a compiler environment that does not support in-place array operations, the model instance can be correctly compiled and run, improving the compatibility of the first compiler and the compilability of the model instance.

[0097] In one possible implementation, the large language model training framework trains the model instance based on the static graph for each batch of training data with the same data volume in the training data; when the data volume of the last batch of training data is less than the data volume of the previous batches, the large language model training framework disables the recompilation operation and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

[0098] In the large language model training framework, training data is divided into multiple batches, each containing a certain number of data samples. When the data volume of these batches is the same, the framework uses a pre-compiled static graph to perform training on these batches.

[0099] For example, suppose the total training data set consists of 40 samples, with each batch containing 4 samples. In the first 10 batches, each batch has the same number of samples (4 samples). For these 10 batches, the training framework uses the same static graph for training. This static graph is compiled and optimized before training begins based on the model's structure and the expected batch size, and then used for training on each batch.

[0100] Since static graphs are optimized for specific amounts of data, in deep learning training, if the model processes inconsistent amounts of data in different batches, it may trigger a compiler recompile. However, such recompile is unnecessary for the last batch with a smaller amount of data. Therefore, large language model training frameworks can prevent recompiling for the last batch.

[0101] Therefore, when training reaches the last batch, and the amount of data in this batch is smaller than that in previous batches, the large language model training framework can disable the recompilation operation to avoid recompilation problems caused by the difference in data volume.

[0102] Continuing the previous example, if there are a total of 42 samples, and each batch typically contains 4 samples, then the first 10 batches will proceed normally, but the last batch will only contain 2 samples. In this case, the training framework will detect that the last batch has less data than the previous batches and disable any operations that might trigger recompilation. For example, in PyTorch, you can disable dynamic graph recompilation by setting the parameter `torch._dynamo.config.automatic_dynamic_shapes=False`, so the last batch can be trained using the previous dynamic graph instead of attempting to compile a new static graph.

[0103] In this embodiment, by using an optimized static graph for training on batches with the same amount of data, and disabling unnecessary recompilation operations in the last batch with insufficient data, compilation errors or performance degradation caused by changes in data volume are avoided. This effectively reduces interruptions and errors during training, reduces unnecessary compilation overhead, accelerates the overall training process, and improves training efficiency. Furthermore, by flexibly switching between static and dynamic graphs, training can proceed smoothly even when the amount of data changes.

[0104] As an embodiment of this disclosure, the step of calling the first compiler to construct a static graph of the model instance includes:

[0105] The user inputs model instances into the large language model training framework and the compilation parameters that instruct the first compiler;

[0106] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests the compilation of the model instance.

[0107] The static graph optimization compiler calls the first compiler to compile the model instance based on the compilation parameters.

[0108] The first compiler captures the bytecode generated when a model instance defined in the large language model training framework is executed;

[0109] The first compiler compiles the captured bytecode to construct the static computation graph of the model. During the compilation process, when the first compiler identifies modules in the model instance that do not support compilation into a static graph, it will skip the compilation operation for these modules. When the first compiler identifies modules in the model instance that do not support compilation operations, it will replace these modules with functionally equivalent modules and perform compilation operations on the code of the replacement equivalent modules, so that these operations can be correctly compiled and executed without changing the functionality of the model.

[0110] The first compiler returns the compiled static graph to the large language model training framework for subsequent training operations.

[0111] The large language model training framework receives the returned static graph and trains the model instances based on the static graph for each batch of training data with the same amount of data in the training data.

[0112] The large language model training framework disables recompilation when the amount of data in the last batch of training data is less than the amount of data in the previous batches, and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

[0113] In one possible implementation, where the compilation backend indicated by the compilation parameters includes a second compiler for operator transformation, the model static graph optimization compiler calls the second compiler to perform a compilation operation; the second compiler captures the computation graphs of forward and backward propagation of the model instance and generates a joint computation graph; the second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the custom operator is encapsulated into the operation of the model static graph optimization compiler using the port of the model static graph optimization compiler.

[0114] During the compilation of model instances, multiple compilation paths and optimization strategies may exist. By setting compilation parameters, a suitable backend compiler can be selected to execute specific compilation tasks. When the compilation backend indicated by the user-input compilation parameters includes a second compiler for operator transformation, the model static graph optimization compiler will call the second compiler to perform compilation operations to transform the model using operators.

[0115] The second compiler captures the computation graphs of model instances during forward and backward propagation and merges these two graphs into a joint computation graph for unified processing. During model training, forward propagation is used to compute the model's output, while backward propagation is used to compute gradients based on that output to update model parameters. Merging them into a joint computation graph optimizes the overall computational flow, reduces memory usage, and improves execution efficiency. The second compiler can capture these two computation graphs and generate the joint computation graph for subsequent optimization and transformation.

[0116] Furthermore, the second compiler converts operators in the joint computation graph into a format executable by the underlying kernel (such as operators at the Aten / Prim level) to enable efficient computation. Operator lowering is the process of converting high-level operators into low-level instruction sets executable by the underlying hardware. In this process, the second compiler analyzes each operator in the joint computation graph and, based on the characteristics and optimization requirements of the underlying kernel, converts the operators into corresponding low-level operators. These low-level operators (such as operators at the Aten / Prim level) have a tighter interface with the underlying hardware, enabling more efficient execution of computational tasks.

[0117] For example, in PyTorch, after the second compiler generates the joint computation graph, this graph is further lowered to the Aten / Prim operator level. These operators are then dispatched by the dispatcher to specific kernel functions for execution based on the attributes of the input tensor processing module (tensor), such as data type and dimension. For instance, a complex matrix multiplication operator is transformed into a series of basic matrix operations, which are ultimately executed on the graphics processing unit (GPU).

[0118] In deep learning models, custom operators may be used to meet specific computational needs. These custom operators are often not natively supported by the model's static graph optimizer and therefore require special processing. During operator transformation, a second compiler detects these custom operators and encapsulates them into internal operations of the model's static graph optimizer using its functionality. This allows these custom operators to be correctly executed and optimized in subsequent computations.

[0119] In other words, when a custom operator is identified in the joint computation graph, the second compiler uses the port of the model static graph optimizer to encapsulate it as an internal compiler operation in order to correctly handle these non-standard operators.

[0120] Take the custom kernel operator `fused_rms_norm` in PyTorch as an example. This is a custom kernel operator bound through the lightweight library pybind. Since second-party compilers cannot directly convert such pybind-based custom kernel functions, it's necessary to register it as an internal PyTorch operation using the `torch.library` API. This way, during operator conversion, the `fused_rms_norm` operator can be treated as a standard torch operation, thus avoiding errors and ensuring computational correctness.

[0121] In this embodiment, by performing operator transformation operations on the computation graph through a second compiler and encapsulating custom operators into the operations of the model static graph optimization compiler, the flexibility and compatibility of the model static graph optimization compiler are significantly improved, enabling it to adapt to large language model training frameworks. By capturing and generating a joint computation graph, performing operator transformations, and adapting to the underlying kernel, the training task can be executed efficiently on the target hardware.

[0122] In one possible implementation, if the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform a compilation operation; the third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, obtaining a hardware-adapted static graph; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped; and deep-level optimization is performed on the hardware-adapted static graph to obtain a deeply optimized static graph.

[0123] When the compilation backend indicated by the user-input compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform compilation operations to deeply optimize the model.

[0124] The third compiler can convert operators adapted to the underlying kernel into operators adapted to the target hardware. During the conversion process, the third compiler will identify each operator in the static graph and convert them from one internal representation (such as Torch tensor processing operations) to another representation that is closer to the hardware operation (such as Triton IR) to adapt to the hardware, reduce the overhead of model instance training, and improve execution efficiency.

[0125] During the conversion process, if a type cannot be directly converted into a hardware-compatible operator, it is skipped. Then, the converted static graph is optimized at a deeper level to improve execution efficiency. Although the skipped, unconvertible operators are not converted into hardware-compatible operators, they can still be used for subsequent training processes with operators adapted to the underlying kernel, thus not affecting model training.

[0126] Deep-level optimization may include various techniques such as operator fusion, constant folding, and traversal optimization, aiming to reduce redundant operations in the computation graph, improve parallelism and data locality, thereby improving overall computational performance.

[0127] For example, when the third compiler encounters a data type conversion operation `self.word_embeddings = self.word_embeddings.to(self.params_dtype)` during operator transformation, it can skip this step because this operation does not support conversion to a representation closer to hardware operations (Triton IR). The third compiler then continues with deep optimization, which may include operator fusion, constant folding, etc., to generate a more efficient computation graph. Further, these optimized computation graphs are transformed by operators to a Low Level Virtual Machine IR (LLVM IR) representation, generating deeply optimized kernel functions for use during actual training. During actual training, the generated static graph calls the generated kernel functions to complete the training.

[0128] In this embodiment, a third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, enabling the model to be flexibly deployed according to the needs of different hardware platforms. Furthermore, through deep-level optimization, redundant operations in the computation graph are reduced, improving parallelism and data locality, thereby enhancing the model's execution efficiency. For operator types that cannot be directly converted, a skipping method ensures the smooth progress of the compilation process, avoiding overall compilation failure due to a single operator issue and enhancing the system's robustness.

[0129] If the user specifies that the compilation process should be performed using a first compiler, a second compiler, and a third compiler when inputting compilation parameters, then one implementation method provided by this disclosure is as follows:

[0130] Input phase:

[0131] User definition and input: Users use predefined model instances by specifying configurations. Furthermore, users specify the compiler (first, second, or third compiler) through compilation parameters.

[0132] Compilation stage:

[0133] Torch compilation and encapsulation: User-submitted model instances are encapsulated through the torch.compile API, which serves as a front-end interface and triggers the subsequent compilation process.

[0134] The first compiler performs bytecode analysis: The first compiler receives Python bytecode (i.e., the Python language implementation of the model instance) and parses the Python code object (PyCodeObject) from the Python stack frame object (PythonFrameObject) of the Python runtime environment.

[0135] The first compiler constructs a functional expression graph (FX graph) based on the parsed bytecode in the PyCodeObject, and inserts a guard mechanism in the process to support dynamic code optimization. Simultaneously, it performs a preliminary compilation of the code.

[0136] The compiled code replaces the original Python bytecode, generating a converted PyCodeObject.

[0137] The second compiler uses the __torch_dispatch__ mechanism to trace the forward and backward propagation processes and generate a joint computation graph that includes forward and backward propagation.

[0138] The second compiler further transforms the joint computation graph to the Aten / Prim operator level, and then the dispatcher distributes the operators to the appropriate kernel functions for execution based on the properties of the input tensor processing module.

[0139] The third compiler receives the graph generated by the second compiler and progressively transforms it into an intermediate representation (Triton IR). Next, the Triton IR is transformed into a multi-level intermediate representation (MLIR), which performs deep optimizations on the computation graph, including operator fusion, constant folding, and traversal optimization. Finally, the MLIR is transformed into LLVM IR by operators, generating deeply optimized kernel functions.

[0140] Model deployment and training phases:

[0141] After the third compiler completes all transformations and optimizations, it generates and returns a static graph to the large language model training framework. Operations in this static graph directly call the deeply optimized kernel functions.

[0142] The large language model training framework uses PyTorch's Distributed Data Parallel (DDP) interface to encapsulate the compiled static graph and begin the first round of data training.

[0143] When processing subsequent batches of data, the guard mechanism determines whether recompilation is necessary. If recompilation is not required, the already compiled code is read directly from the cache; otherwise, the recompilation process is triggered.

[0144] The training data is forward-propagated through the forward static graph of the static graph to obtain the prediction results. Then, the loss value is calculated based on the prediction results and the labels of the training data. Based on the loss value, the gradient value of each parameter is calculated using the chain rule of calculus through the backward static graph of the static graph.

[0145] After the gradient values ​​are input into the optimizer, the model parameters are updated. Then, the next round of data iteration begins, repeating the forward propagation, backpropagation, and parameter update steps to train the model instance.

[0146] Compared to the traditional dynamic graph model, the implementation method provided in this disclosure improves computational efficiency by approximately 15%.

[0147] In one possible implementation, when the compilation backend indicated by the compilation parameters includes a fourth compiler adapted to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform compilation operations; the fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform; wherein, when a set communication operator for integrating the processing results of multiple processors is identified, the function of the set communication operator in the backpropagation computation graph is converted to: retrieving the set communication results saved in the forward propagation stage from memory; and performing deep-level optimization on the static graph adapted to the hardware to obtain a deeply optimized static graph.

[0148] In the development and deployment of deep learning models, different hardware platforms (such as GPUs) may require different compilation strategies to optimize hardware performance. In this implementation, when the compilation parameters indicate that the compilation backend includes a fourth compiler adapted for a specific hardware platform, the model static graph optimizer will call the fourth compiler adapted for that hardware platform according to the compilation parameters, so that the compiled model can run efficiently on that hardware.

[0149] Operators are the basic operational units in a model instance, and different hardware platforms have different requirements for the implementation of operators. During the compilation process, the fourth compiler can convert operators originally adapted to the underlying kernel into operators adapted to specific hardware platforms.

[0150] During the training process of deep learning, ensemble communication operators are used to integrate computational results from different processors (such as GPUs).

[0151] To avoid repeatedly executing set communication operations during the backpropagation phase, this implementation stores the set communication results already completed during the forward propagation phase in memory and retrieves these results directly from memory during backpropagation. Furthermore, the static graph adapted to the hardware is deeply optimized to further improve computational efficiency.

[0152] When using the fourth compiler for adaptation, the fourth compiler will recognize the use of this operator and optimize it to calculate and save the set communication results in the forward propagation phase, and directly retrieve these results from memory in the back propagation phase, thus avoiding the GPU lockup problem that may be caused by re-executing the set communication operation.

[0153] In addition, the fourth compiler will perform in-depth optimizations on the static graph, such as adjusting the computation order and merging similar operations, to further improve the model's running efficiency.

[0154] In this embodiment, by selecting and invoking a fourth compiler optimized for a specific hardware platform, efficient compilation and deployment of deep learning models are achieved. Furthermore, by optimizing the use of the set communication operator, performance issues (such as GPU deadlock) that might arise from repeatedly executing set communication operations during the backpropagation phase are avoided. This improves the efficiency and stability of training model instances on specific hardware platforms.

[0155] Compared to the traditional dynamic graph model, the implementation method provided in this disclosure improves computational efficiency by about 10%.

[0156] In one possible implementation, the model training system can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The system can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the system can be implemented by a server.

[0157] In addition, this disclosure also provides model training methods, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the model training systems provided in this disclosure. The corresponding technical solutions and descriptions can be found in the relevant records in the method section, and will not be repeated here.

[0158] Figure 2A flowchart illustrating a model training method according to an embodiment of this disclosure, applied to the above-described system, is shown. Figure 2 As shown, the method includes:

[0159] In step S21, the large language model training framework receives the model instance and compilation parameters input by the user;

[0160] In step S22, the large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests to compile the model instance.

[0161] In step S23, the model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operation on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework.

[0162] In step S24, the large language model training framework receives the static graph and trains the model instance based on the static graph.

[0163] In one possible implementation, the graph compilation operation on the model instance includes:

[0164] When the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform the compilation operation;

[0165] The first compiler obtains the bytecode of the model instance;

[0166] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies a module in the model instance that does not support compilation into a static graph, it skips the compilation operation for the module that does not support compilation into a static graph; when the first compiler identifies a module in the model instance that it does not support compilation, it replaces the module that does not support compilation with a functionally equivalent module and performs a compilation operation on the code of the equivalent module.

[0167] In one possible implementation, the module that does not support compilation to a static graph includes a recomputation module, and the graph compilation operation on the model instance includes:

[0168] When the first compiler identifies a recomputation module of the model instance, it skips the compilation operation for the recomputation module.

[0169] In one possible implementation, the first compiler does not support modules that perform compilation operations, including in-place array operation modules. The graph compilation operation on the model instance includes:

[0170] When the code of the in-place array operation module used for in-place array operations is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and the code of the non-in-place array operation module is compiled.

[0171] In one possible implementation, training the model instance based on the static graph includes:

[0172] The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data.

[0173] The large language model training framework disables recompilation when the amount of data in the last batch of training data is less than the amount of data in the previous batches, and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

[0174] In one possible implementation, the graph compilation operation on the model instance includes:

[0175] When the compilation backend indicated by the compilation parameters includes a second compiler for operator transformation, the model static graph optimization compiler calls the second compiler to perform the compilation operation.

[0176] The second compiler captures the computation graphs of forward and backward propagation of model instances and generates a joint computation graph;

[0177] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator is identified in the large language model training framework, the custom operator is encapsulated into the operation of the model static graph optimizer using the port of the model static graph optimizer.

[0178] In one possible implementation, the graph compilation operation on the model instance includes:

[0179] When the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform the compilation operation;

[0180] The third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, thus obtaining a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped.

[0181] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0182] In one possible implementation, the graph compilation operation on the model instance includes:

[0183] When the compilation backend indicated by the compilation parameters includes a fourth compiler adapted for a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform the compilation operation;

[0184] The fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform; wherein, when a set communication operator for integrating the processing results of multiple processors is identified, the function of the set communication operator in the backpropagation computation graph is converted into: retrieving the set communication results saved in the forward propagation stage from memory;

[0185] The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

[0186] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0187] In some embodiments, the functions or modules of the methods provided in this disclosure can be used to implement the system described in the above system embodiments. The specific implementation can be referred to the description of the above system embodiments, and for the sake of brevity, it will not be repeated here.

[0188] This disclosure also proposes a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the aforementioned system. The computer-readable storage medium can be volatile or non-volatile.

[0189] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to implement the above-described system.

[0190] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device implements the above-described system.

[0191] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0192] Figure 3 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0193] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0194] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0195] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0196] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0197] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0198] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0199] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0200] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0201] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0203] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0204] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0205] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0206] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0207] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model training system, characterized in that, include: A large language model training framework that receives model instances and compilation parameters from the user; The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests the compilation of the model instance. The model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operations on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework; wherein, the compilation backend includes a fourth compiler for adapting to a specific hardware platform, when the fourth compiler recognizes the set communication operator used to integrate the processing results of multiple processors, the function of the set communication operator in the backpropagation computation graph is converted to: obtaining the set communication results saved in the forward propagation stage from memory, and performing deep-level optimization on the static graph; The large language model training framework receives the static graph and trains the model instance based on the static graph.

2. The system according to claim 1, characterized in that, When the compilation backend indicated by the compilation parameters includes a first compiler for capturing the forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform the compilation operation; The first compiler obtains the bytecode of the model instance; The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies a module in the model instance that does not support compilation into a static graph, it skips the compilation operation for the module that does not support compilation into a static graph; when the first compiler identifies a module in the model instance that it does not support compilation, it replaces the module that does not support compilation with a functionally equivalent module and performs a compilation operation on the code of the equivalent module.

3. The system according to claim 2, characterized in that, The module that does not support compilation into a static graph includes a recomputation module. When the first compiler identifies a recomputation module of the model instance, it skips the compilation operation of the recomputation module. The first compiler does not support modules that perform compilation operations, including in-place array operation modules. When the code of an in-place array operation module used for in-place array operations is identified, the code of the in-place array operation module is replaced with the code of a non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module.

4. The system according to claim 1, characterized in that, The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data. The large language model training framework disables recompilation when the amount of data in the last batch of training data is less than the amount of data in the previous batches, and uses the training data of the last batch to train the model instance based on the dynamic graph of the model instance.

5. The system according to claim 1, characterized in that, When the compilation backend indicated by the compilation parameters includes a second compiler for operator transformation, the model static graph optimization compiler calls the second compiler to perform the compilation operation. The second compiler captures the computation graphs of forward and backward propagation of model instances and generates a joint computation graph; The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the custom operator is encapsulated into the operation of the model static graph optimizer using the port of the model static graph optimizer.

6. The system according to claim 1, characterized in that, When the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computation graph, the model static graph optimization compiler calls the third compiler to perform the compilation operation; The third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware, thus obtaining a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped. The static graph of the adapted hardware is optimized at a deep level to obtain a deeply optimized static graph.

7. The system according to claim 6, characterized in that, When the compilation backend indicated by the compilation parameters includes a fourth compiler adapted for a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform the compilation operation; The fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform.

8. A model training method, characterized in that, The system applied to any one of claims 1 to 7 comprises: A large language model training framework that receives model instances and compilation parameters from the user; The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler and requests the compilation of the model instance. The model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform graph compilation operations on the model instance, obtains the static graph of the model instance, and returns it to the large language model training framework; wherein, the compilation backend includes a fourth compiler for adapting to a specific hardware platform, when the fourth compiler recognizes the set communication operator used to integrate the processing results of multiple processors, the function of the set communication operator in the backpropagation computation graph is converted to: obtaining the set communication results saved in the forward propagation stage from memory, and performing deep-level optimization on the static graph; The large language model training framework receives the static graph and trains the model instance based on the static graph.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to implement the system according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the system according to any one of claims 1 to 7.

Images