Model training system and method, storage medium and electronic equipment

Abstract

The invention relates to a model training system and method, a storage medium and electronic equipment, and the system comprises a large language model training framework which is used for receiving a model instance and compiling parameters input by a user; the large language model training framework transmits the model instance and compiling parameters into a port of a model static graph optimization compiler, and requests to compile the model instance; the model static graph optimization compiler is used for receiving an input model instance and compiling parameters, performing graph compiling operation on the model instance by utilizing a compiling rear end corresponding to the compiling parameters to obtain a static graph of the model instance, and returning the static graph to the large language model training framework; and the large language model training framework receives the static graph and trains the model instance based on the static graph. The embodiment of the invention can improve the training efficiency and the resource utilization rate.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a model training system and method, a storage medium and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly used in various fields. These models have achieved powerful natural language processing capabilities based on deep learning technology. However, the training process of large language models requires processing massive amounts of data and complex model parameters, which places extremely high demands on computing resources.

[0003] The large language model training framework is a framework for training large language models. It aims to improve the efficiency and scalability of large-scale model training. It improves the efficiency of large-scale model training through various technical means such as mixed precision and data parallelism.

[0004] However, although large language model training frameworks have made significant progress in training efficiency, as the model size increases, the processing resources and costs consumed by training large models remain high. Summary of the invention

[0005] The present invention proposes a model training technology solution.

[0006] According to one aspect of the present disclosure, a model training system is provided, comprising:

[0007] Large language model training framework, receiving model instances and compilation parameters input by users;

[0008] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting compilation of the model instance;

[0009] The model static graph optimization compiler receives the incoming model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework;

[0010] The large language model training framework receives the static graph and trains the model instance based on the static graph.

[0011] In a possible implementation, when the compilation backend indicated by the compilation parameter includes a first compiler for capturing a forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation;

[0012] The first compiler obtains the bytecode of the model instance;

[0013] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler recognizes that there are modules in the model instance that do not support compilation into a static graph, the first compiler skips the compilation operation on the modules that do not support compilation into a static graph; when the first compiler recognizes that there are modules in the model instance that the first compiler does not support compilation operations, the first compiler replaces the modules that do not support compilation operations with equivalent modules with equivalent functions, and performs compilation operations on the codes of the equivalent modules.

[0014] In a possible implementation, the module that does not support compilation into a static graph includes a recalculation module, and the first compiler, when identifying the recalculation module of the model instance, skips the compilation operation of the recalculation module;

[0015] The modules that the first compiler does not support performing compilation operations include in-place array operation modules. When the code of the in-place array operation module used to perform in-place array operations is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module.

[0016] In a possible implementation, the large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data;

[0017] The large language model training framework disables recompilation operations when the amount of data in the last batch of training data is less than the amount of data in the previous batch, and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

[0018] In a possible implementation, when the compilation backend indicated by the compilation parameter includes a second compiler for operator conversion, the model static graph optimization compiler calls the second compiler to perform a compilation operation;

[0019] The second compiler captures the forward propagation and reverse propagation computation graphs of the model instance and generates a joint computation graph;

[0020] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the port of the model static graph optimization compiler is used to encapsulate the custom operator into the operation of the model static graph optimization compiler.

[0021] In a possible implementation, when the compilation backend indicated by the compilation parameter includes a third compiler for performing deep-level optimization on the computation graph, the model static graph optimization compiler calls the third compiler to perform a compilation operation;

[0022] The third compiler converts the operator adapted to the underlying kernel into an operator adapted to the hardware, and obtains a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped;

[0023] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0024] In a possible implementation, when the compilation backend indicated by the compilation parameter includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform a compilation operation;

[0025] The fourth compiler converts the operator adapted to the underlying kernel into an operator adapted to the specific hardware platform; wherein, when a collective communication operator for integrating processing results of multiple processors is identified, the function of the collective communication operator in the back propagation calculation graph is converted to: obtaining the collective communication result saved in the forward propagation stage from the memory;

[0026] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0027] According to one aspect of the present disclosure, a model training method is provided, which is applied to the above system, including:

[0028] Large language model training framework, receiving model instances and compilation parameters input by users;

[0029] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting compilation of the model instance;

[0030] The model static graph optimization compiler receives the incoming model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework;

[0031] The large language model training framework receives the static graph and trains the model instance based on the static graph.

[0032] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0033] In a case where the compilation backend indicated by the compilation parameter includes a first compiler for capturing a forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation;

[0034] The first compiler obtains the bytecode of the model instance;

[0035] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler recognizes that there are modules in the model instance that do not support compilation into a static graph, the first compiler skips the compilation operation on the modules that do not support compilation into a static graph; when the first compiler recognizes that there are modules in the model instance that the first compiler does not support compilation operations, the first compiler replaces the modules that do not support compilation operations with equivalent modules with equivalent functions, and performs compilation operations on the codes of the equivalent modules.

[0036] In a possible implementation, the module that does not support compilation into a static graph includes a recalculation module, and the graph compilation operation on the model instance includes:

[0037] The first compiler, when identifying the recalculation module of the model instance, skips the compilation operation of the recalculation module;

[0038] In a possible implementation, the module that the first compiler does not support for performing compilation operations includes an in-place array operation module, and the graph compilation operation on the model instance includes:

[0039] When the code of the in-place array operation module for performing the in-place array operation is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and a compile operation is performed on the code of the non-in-place array operation module.

[0040] In a possible implementation, the training the model instance based on the static graph includes:

[0041] The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data;

[0042] The large language model training framework disables recompilation operations when the amount of data in the last batch of training data is less than the amount of data in the previous batch, and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

[0043] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0044] In a case where the compilation backend indicated by the compilation parameter includes a second compiler for operator conversion, the model static graph optimization compiler calls the second compiler to perform a compilation operation;

[0045] The second compiler captures the forward propagation and reverse propagation computation graphs of the model instance and generates a joint computation graph;

[0046] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the port of the model static graph optimization compiler is used to encapsulate the custom operator into the operation of the model static graph optimization compiler.

[0047] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0048] In a case where the compilation backend indicated by the compilation parameters includes a third compiler for performing deep-level optimization on the computation graph, the model static graph optimization compiler calls the third compiler to perform a compilation operation;

[0049] The third compiler converts the operator adapted to the underlying kernel into an operator adapted to the hardware, and obtains a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped;

[0050] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0051] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0052] In a case where the compilation backend indicated by the compilation parameters includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform a compilation operation;

[0053] The fourth compiler converts the operator adapted to the underlying kernel into an operator adapted to the specific hardware platform; wherein, when a collective communication operator for integrating processing results of multiple processors is identified, the function of the collective communication operator in the back propagation calculation graph is converted to: obtaining the collective communication result saved in the forward propagation stage from the memory;

[0054] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0055] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0056] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0057] In the disclosed embodiment, the large language model training framework receives the model instance and compilation parameters input by the user, and then passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting to compile the model instance; the model static graph optimization compiler receives the passed-in model instance and compilation parameters, and uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework; the large language model training framework receives the static graph, and trains the model instance based on the static graph. Therefore, by introducing a static graph optimization compiler into the large language model training framework, since the static graph can remove the redundant calculations and storage of intermediate variables commonly seen in the dynamic graph, thereby reducing the computation and memory requirements during training and reasoning, hardware resources can be used more efficiently, unnecessary computation and memory overhead can be reduced during the training process, and training efficiency and resource utilization can be improved.

[0058] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0060] Figure 1 A block diagram of a model training system according to an embodiment of the present disclosure is shown.

[0061] Figure 2 A flowchart of a model training method according to an embodiment of the present disclosure is shown.

[0062] Figure 3 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0064] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0065] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0066] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0067] Figure 1 A block diagram of a model training system according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the system comprises:

[0068] The large language model training framework 101 receives the model instance and compilation parameters input by the user;

[0069] The large language model training framework 101 passes the model instance and compilation parameters to the port of the model static graph optimization compiler 102 to request compilation of the model instance;

[0070] The model static graph optimization compiler 102 receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework;

[0071] The large language model training framework 101 receives the static graph and trains the model instance based on the static graph.

[0072] The large language model training framework can be software or a system used to train large natural language processing models. The large language model training framework provides the infrastructure and tools required to train large-scale language models to achieve the construction and training of large language models.

[0073] The large language model training framework may include multiple components, such as a model definition interface, data preprocessing tools, an optimization algorithm library, a distributed training component, etc. In one example, the large language model training framework may be used to train a large transformer model Transformer, which can support multiple parallel strategies (such as data parallelism, model parallelism, pipeline parallelism) and mixed precision training, and thus can utilize multiple GPUs or multi-node clusters to efficiently train large language models.

[0074] The model instance can be a large language model to be trained. The model instance can define functional modules and parameters based on the large language model training framework to achieve specific functions. The model instance can be a neural network model. The model instance is the object operated by the large language model training framework. The large language model can calculate the prediction results through the forward propagation of the large language model according to the input training data and training objectives, and then update the weight parameters through back propagation to achieve the training of the large language model.

[0075] The large language model training framework can receive model instances and compilation parameters input by users. Users can provide the model instances and compilation parameters to be trained to the large language model training framework through the user graphical interface or interface provided by the large language model training framework, so that users can flexibly select models and compilation strategies to adapt to different training requirements and hardware environments, thereby improving training efficiency and effectiveness.

[0076] The model static graph optimization compiler is used to generate an optimized model representation (i.e., static graph) through static graph analysis and optimization technology to improve the execution efficiency and resource utilization of the model. The static graph optimization compiler can call the corresponding compiler to perform compilation operations based on the compilation parameters entered by the user, analyze the structure and parameters of the model instance, and generate a static graph.

[0077] The compilation parameters may be parameters used to control the model compilation behavior, which define the optimization strategy, compilation backend, and possible other settings used in the compilation process. The compilation backend may include: a first compiler for capturing the forward propagation computation graph; a second compiler for performing operator conversion; and a third compiler for performing deep level optimization of the computation graph. For specific implementations, please refer to the possible implementations provided in the present disclosure, which will not be described in detail here.

[0078] The large language model training framework passes the model instance and compilation parameters provided by the user to the model static graph optimization compiler (such as Torch Compile) to request the model to be compiled to generate an optimized static graph representation. The static graph optimization compiler is used to generate an optimized static graph based on the model instance and compilation parameters. By introducing the static graph optimization compiler, the model can be optimized before training, which improves the computational efficiency and resource utilization during the large language model training process to reduce computational redundancy and memory utilization.

[0079] The model static graph optimization compiler performs graph compilation operations based on the received model instance and compilation parameters, generates an optimized static graph representation, and returns it to the large language model training framework. When performing graph compilation operations, the compilation backend analyzes and converts the model instance, analyzes the structure and properties of the model instance, and then optimizes the large language model according to optimization rules (such as operator fusion, memory reuse, parallel execution, etc.) to generate a more efficient and resource-saving static graph representation. Then, the compilation backend returns the optimized static graph to the large language model training framework for subsequent training operations.

[0080] After receiving the static graph, the large language model training framework will train the model instance based on the static graph. The training framework will parse the structure and parameters of the static graph and perform training according to the calculation process and data flow defined in the graph. For example, after receiving the optimized static graph of the model instance returned by Torch Compile, the large language model training framework will perform training according to the optimized calculation process and data flow defined in the static graph, thereby significantly improving the training speed and efficiency.

[0081] In the disclosed embodiment, the large language model training framework receives the model instance and compilation parameters input by the user, and then passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting to compile the model instance; the model static graph optimization compiler receives the passed-in model instance and compilation parameters, and uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework; the large language model training framework receives the static graph, and trains the model instance based on the static graph. Therefore, by introducing a static graph optimization compiler into the large language model training framework, since the static graph can remove the redundant calculations and storage of intermediate variables commonly seen in the dynamic graph, thereby reducing the computation and memory requirements during training and reasoning, hardware resources can be used more efficiently, unnecessary computation and memory overhead can be reduced during the training process, and training efficiency and resource utilization can be improved.

[0082] In a possible implementation, when the compilation backend indicated by the compilation parameters includes a first compiler for capturing a forward propagation computational graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation; the first compiler obtains the bytecode of the model instance; the first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler identifies that there are modules in the model instance that do not support compilation into a static graph, the compilation operation on the modules that do not support compilation into a static graph is skipped; when the first compiler identifies that there are modules in the model instance that the first compiler does not support compilation operations, the first compiler replaces the modules that do not support compilation operations with functionally equivalent modules, and performs compilation operations on the code of the equivalent modules.

[0083] The user specifies the compilation backend through compilation parameters. When the specified compilation backend is the first compiler, the model static graph optimization compiler will call the first compiler to capture and process the forward propagation calculation graph of the model. The first compiler can process the forward propagation calculation graph of the model instance to obtain the static graph of the model instance. Exemplarily, the user enters the compilation parameter "aaa" through the command line, and the model static graph optimization compiler recognizes that the compilation backend corresponding to the parameter is the first compiler, then it will call the first compiler to capture and process the forward propagation calculation graph of the model, and compile the model instance to generate a static graph. Convert the original dynamic graph model to a static graph model.

[0084] When performing a compilation operation, the first compiler obtains the bytecode of the model instance, that is, extracts the bytecode required to perform the compilation operation from the model instance. The bytecode is a low-level representation of the model instance when it is executed in the computer memory. The first compiler obtains the structure and calculation process of the model by parsing the bytecode.

[0085] Specifically, the compiler analyzes the operations and data flow in the bytecode, identifies the forward propagation path of the model, and converts this information into a static graph structure through optimization and reconstruction for subsequent training and reasoning.

[0086] Since there are modules in the model instance that do not support compilation into static graphs, during the compilation process, when it is identified that there are modules in the model instance that do not support compilation into static graphs, the compilation operation of the modules that do not support compilation into static graphs can be skipped. In other words, the compilation operation is not performed on the modules that do not support compilation into static graphs to avoid compilation errors. The modules that do not support compilation into static graphs can still be executed in the form of dynamic graphs.

[0087] In addition, when it is identified that there are modules in the model instance that the first compiler does not support the compilation operation, in order to improve the training efficiency and resource utilization without affecting the complete function of the model, the modules that do not support the compilation operation can be replaced with equivalent modules with equivalent functions, and the compilation operation is performed on the code of the equivalent module. The equivalent module implements the same function as the module that the first compiler does not support the compilation operation, and there may be differences in the implementation process, but it does not affect the training effect of the large model.

[0088] After replacing the modules that the first compiler does not support to perform compilation operations with other operations that are functionally equivalent, the first compiler compiles the replaced operations to obtain a static graph of the model instance.

[0089] In an embodiment of the present disclosure, when the compilation backend indicated by the compilation parameters includes a first compiler for capturing a forward propagation computational graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation; the first compiler obtains the bytecode of the model instance; the first compiler compiles the bytecode to construct a static graph of the model. Since the model instance is defined and input based on a large language model training framework, there may be a situation where the model static graph optimization compiler is incompatible. Then, when a module that does not support compilation into a static graph is identified in the model instance, the compilation operation of the module that does not support compilation into a static graph can be skipped; and when a module that the first compiler does not support the execution of a compilation operation is identified in the model instance, the module that does not support the execution of a compilation operation is replaced with an equivalent module with equivalent functions, and the code of the equivalent module is compiled. Therefore, adaptation to the large language model training framework and the static graph optimization compiler is achieved. The adapted large language model training framework can train model instances in static graph mode. Through graph compilation and graph optimization technology, it can more effectively utilize GPU resources and reduce runtime overhead, thereby improving training efficiency. Compared with the dynamic graph mode, the overall training efficiency is improved by about 15%.

[0090] In one possible implementation, the modules that do not support compilation into static graphs include recalculation modules, and the first compiler, when identifying the recalculation module of the model instance, skips the compilation operation of the recalculation module; the modules that the first compiler does not support compilation operations include in-place array operation modules, and when identifying the code of the in-place array operation module for performing in-place array operations, the first compiler replaces the code of the in-place array operation module with the code of the non-in-place array operation module, and performs compilation operations on the code of the non-in-place array operation module.

[0091] In this implementation, the modules that do not support compilation into static graphs include recalculation modules. Then, when the first compiler recognizes that the model instance contains a recalculation module, since the first compiler cannot convert the recalculation module into a static graph, directly compiling the recalculation module will result in an error. Therefore, the first compiler can skip the compilation of the recalculation module. The recalculation module is used to determine which layers in the model instance should have their activation values ​​recalculated to strike a balance between memory usage and computational efficiency. This requires the ability to dynamically build and modify the computation graph, which is feasible in the dynamic graph execution mode. Therefore, the recalculation module cannot be converted into a static graph. The specific execution process of the recalculation module is not described in detail in this disclosure.

[0092] For example, in the related art, the recalculation mechanism is implemented by calling the application programming interface (API) of the tool function torch.utils.checkpoint of the open source machine learning library PyTorch to reduce memory usage. However, when the first compiler attempts to compile a model containing these recalculation modules, an error message will be displayed. Therefore, in this implementation, the compilation of the recalculation module can be skipped.

[0093] In this implementation, the modules that the first compiler does not support for executing compilation operations include in-place array operation modules. Then, when the first compiler identifies an in-place array operation module, it converts it into a non-in-place array operation module and then executes the compilation operation.

[0094] In-place array operations are operations that directly modify the original array or memory space. Such operations are not supported in the first compiler because they may destroy the consistency of the data. Therefore, when an in-place array operation module is identified, the first compiler will convert it into an equivalent form that operates on the new memory space (i.e., non-in-place array operations), thereby avoiding compilation errors and ensuring the correctness of the data.

[0095] For example, in the related art, there may be in-place array operations. However, the first compiler does not support such in-place array operations, so it converts them into non-in-place array operations, so that the calculation results are stored in the new memory space, thereby avoiding potential compilation errors and data consistency issues.

[0096] In the disclosed embodiment, the module that does not support compilation into a static graph includes a recalculation module. When the first compiler identifies the recalculation module of the model instance, it skips the compilation operation on the recalculation module. The module that the first compiler does not support the compilation operation includes an in-place array operation module. When the code of the in-place array operation module is identified, the code of the in-place array operation module is replaced with the code of a non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module. Thus, by skipping the compilation of the recalculation module that does not support compilation into a static graph, compilation errors caused by the compiler not supporting the module are avoided; by converting the in-place array operation module into a non-in-place array operation, it is ensured that the model instance can be correctly compiled and run even in a compiler environment that does not support in-place array operations, thereby improving the compatibility of the first compiler and the compilability of the model instance.

[0097] In a possible implementation, the large language model training framework trains the model instance based on the static graph for each batch of training data with the same data volume in the training data; when the data volume of the last batch of training data is less than the data volume of the previous batch, the large language model training framework disables the recompilation operation and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

[0098] In the large language model training framework, the training data is divided into multiple batches, each batch contains a certain number of data samples. When the amount of data in these batches is the same, the framework uses the compiled static graph to perform training for these batches.

[0099] For example, assume that the total amount of training data is 40 samples, and each batch contains 4 samples. In the first 10 batches, the amount of data in each batch is the same (i.e., 4 samples). For these 10 batches, the training framework will use the same static graph for training. This static graph has been compiled and optimized based on the structure of the model and the expected data batch size before training begins, and then this graph is used for training for each batch.

[0100] Since static graphs are optimized for a specific amount of data, in deep learning training, if the model processes inconsistent amounts of data in different batches, it may trigger a compiler recompilation operation. However, this recompilation is unnecessary in the last batch with a smaller amount of data. Therefore, the large language model training framework can prohibit recompilation of the last batch.

[0101] Then, when the training reaches the last batch and the amount of data in this batch is smaller than that of the previous batch, in order to avoid the recompilation problem caused by the different data amounts, the large language model training framework can disable the recompilation operation.

[0102] Continuing with the previous example, if there are 42 samples in total, and each batch normally contains 4 samples, then the first 10 batches will proceed normally, but the last batch will only contain 2 samples. In this case, the training framework will detect that the amount of data in the last batch is smaller than the previous batches and disable operations that may trigger recompilation. Exemplarily, in PyTorch, you can disable recompilation of dynamic graphs by setting the parameter torch._dynamo.config.automatic_dynamic_shapes=False, so that the last batch can be trained using the previous dynamic graph instead of trying to compile a new static graph.

[0103] In the disclosed embodiment, by using optimized static graphs for training for batches with the same amount of data, and disabling unnecessary recompilation operations in the last batch with insufficient data, compilation errors or performance degradation caused by changes in data volume are avoided. Interruptions and errors in the training process are effectively reduced, unnecessary compilation overhead is reduced, the overall training process is accelerated, and training efficiency is improved. In addition, by flexibly switching between static graphs and dynamic graphs, it is ensured that training can proceed smoothly even when the amount of data changes.

[0104] As an embodiment of the present disclosure, the step of calling the first compiler to construct a static graph of the model instance includes:

[0105] A model instance input by a user through a large language model training framework and a compilation parameter indicating a first compiler;

[0106] The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting compilation of the model instance;

[0107] The model static graph optimization compiler calls the first compiler to compile the model instance according to the compilation parameters;

[0108] The first compiler captures the bytecode generated when a model instance defined in a large language model training framework is executed;

[0109] The first compiler compiles the captured bytecode to construct a static computational graph of the model; during the compilation process, when the first compiler identifies that there are modules in the model instance that do not support compilation into a static graph, the compilation operation of these modules will be skipped; when the first compiler identifies that there are modules in the model instance that it does not support compilation operations, these modules that do not support compilation operations will be replaced with equivalent modules with equivalent functions, and compilation operations will be performed on the codes of the replaced equivalent modules, so that these operations can be correctly compiled and executed without changing the model function;

[0110] The first compiler returns the compiled static graph to the large language model training framework for subsequent training operations;

[0111] The large language model training framework receives the returned static graph and trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data.

[0112] The large language model training framework disables recompilation operations when the amount of data in the last batch of training data is less than the amount of data in the previous batch, and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

[0113] In a possible implementation, when the compilation backend indicated by the compilation parameters includes a second compiler for performing operator conversion, the model static graph optimization compiler calls the second compiler to perform compilation operations; the second compiler captures the forward propagation and backward propagation computation graphs of the model instance to generate a joint computation graph; the second compiler converts the operators in the joint computation graph into operators that are adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the port of the model static graph optimization compiler is used to encapsulate the custom operator as an operation of the model static graph optimization compiler.

[0114] In the process of compiling a model instance, there may be multiple compilation paths and optimization strategies. By setting the compilation parameters, a suitable backend compiler can be selected to perform a specific compilation task. When the compilation backend indicated by the compilation parameters input by the user includes a second compiler for operator conversion, the model static graph optimization compiler will call the second compiler to perform compilation operations to perform operator conversion on the model.

[0115] The second compiler captures the computational graphs of the model instance during forward propagation and backward propagation, and merges the two computational graphs into a joint computational graph for unified processing. During model training, forward propagation is used to calculate the output of the model, while backward propagation is used to calculate gradients based on the output to update model parameters. By merging the two into a joint computational graph, the overall computational process can be optimized, memory usage can be reduced, and execution efficiency can be improved. The second compiler can capture these two computational graphs and generate a joint computational graph for subsequent optimization and conversion.

[0116] Furthermore, the second compiler converts the operators in the joint computation graph into a format that can be executed by the underlying kernel (such as operators at the Aten / Prim level) so that the calculation can be performed efficiently. Operator conversion (Lowering) is the process of converting high-level operators into low-level instruction sets that can be executed by the underlying hardware. In this process, the second compiler analyzes each operator in the joint computation graph and converts the operators into corresponding lower-level operators based on the characteristics and optimization requirements of the underlying kernel. These lower-level operators (such as operators at the Aten / Prim level) have a closer interface with the underlying hardware and can perform computing tasks more efficiently.

[0117] For example, in PyTorch, after the second compiler generates a joint computation graph, the computation graph will be further converted (lowering) to the Aten / Prim operator level. These operators will then be dispatched by the dispatcher to specific kernel functions (kernel) for execution based on the attributes of the input tensor processing module (tensor), such as data type and dimension. For example, a complex matrix multiplication operator will be converted into a series of basic matrix operations, which are ultimately executed on the graphics processor GPU.

[0118] In deep learning models, some custom operators may be used to meet specific computing requirements. These custom operators are often not natively supported by the model static graph optimization compiler, so they need to be processed in a specific way. During the operator conversion process, the second compiler will detect these custom operators and use the functions of the model static graph optimization compiler to encapsulate them as internal operations of the model static graph optimization compiler. This allows these custom operators to be correctly executed and optimized in subsequent calculations.

[0119] That is, when custom operators in the joint computation graph are identified, the second compiler uses the port of the model static graph optimization compiler to encapsulate them as operations inside the compiler so as to correctly handle these non-standard operators.

[0120] Take the custom kernel operator fused_rms_norm in PyTorch as an example. This is a custom kernel operator bound by the lightweight library pybind. Since the second compiler cannot directly convert this pybind-based custom kernel function, it is necessary to use the torch.library API to register it as an internal operation op of PyTorch. In this way, during the operator conversion process, the fused_rms_norm operator can be treated as a standard torch op, thus avoiding errors and ensuring the correctness of the calculation.

[0121] In the disclosed embodiment, the second compiler performs operator conversion operations on the computation graph, and encapsulates the custom operator as the operation of the model static graph optimization compiler, which significantly improves the flexibility and compatibility of the model static graph optimization compiler, so that the model static graph optimization compiler can be adapted to the large language model training framework. By capturing and generating a joint computation graph, performing operator conversion, and adapting the underlying kernel, the training task can be efficiently executed on the target hardware.

[0122] In a possible implementation, when the compilation backend indicated by the compilation parameters includes a third compiler for deep-level optimization of the computational graph, the model static graph optimization compiler calls the third compiler to perform compilation operations; the third compiler converts operators adapted to the underlying kernel into operators adapted to the hardware to obtain a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped; the static graph adapted to the hardware is deeply optimized to obtain a deeply optimized static graph.

[0123] When the compilation backend indicated by the compilation parameters input by the user includes a third compiler for performing deep-level optimization on the computational graph, the model static graph optimization compiler will call the third compiler to perform compilation operations to perform deep optimization on the model.

[0124] The third compiler can convert operators that adapt to the underlying kernel into operators that adapt to the target hardware. During the conversion process, the third compiler will identify each operator in the static graph and convert them from an internal representation (such as Torch's tensor processing operations) to another representation that is closer to hardware operations (such as Triton IR) to adapt to the hardware, reduce the overhead during model instance training, and improve execution efficiency.

[0125] During the conversion process, if you encounter types that cannot be directly converted to hardware-adapted operators, skip these types; then, perform deep optimization on the converted static graph to improve execution efficiency. Although the skipped operators that cannot be converted are not converted to hardware-adapted operators, they can still be used for subsequent training processes as operators that adapt to the underlying kernel, thus not affecting the training of the model.

[0126] Deep-level optimization may include operator fusion, constant folding, traversal optimization and other techniques, which aim to reduce redundant operations in the computational graph, improve parallelism and data locality, and thus improve overall computing performance.

[0127] Exemplarily, when the third compiler encounters the data type conversion operation self.word_embeddings = self.word_embeddings.to (self.params_dtype) during the operator conversion process, since the operation does not support conversion to a representation that is closer to the hardware operation (Triton IR), the third compiler can skip this step. Then, the third compiler continues to perform deep optimization, which may include operations such as operator fusion and constant folding to generate a more efficient calculation graph. Furthermore, these optimized calculation graphs will be converted to a low-level virtual machine intermediate representation (Low Level Virtual Machine IR, LLVM IR) by the operator, and a deeply optimized kernel function will be generated for calling during the actual training process. During the actual training process, the generated static graph will call the generated kernel function to complete the training during execution.

[0128] In the disclosed embodiment, the operator adapted to the underlying kernel is converted into an operator adapted to the hardware through a third compiler, so that the model can be flexibly deployed according to the requirements of different hardware platforms; further, through deep-level optimization, redundant operations in the calculation graph are reduced, parallelism and data locality are improved, thereby improving the execution efficiency of the model. For operator types that cannot be directly converted, the skipping method ensures the smooth progress of the compilation process, avoids the overall compilation failure caused by a single operator problem, and enhances the robustness of the system.

[0129] If the user indicates to use the first compiler, the second compiler and the third compiler for compilation processing at the same time when inputting compilation parameters, then an implementation method provided by the embodiment of the present disclosure is as follows:

[0130] Input stage:

[0131] User definition and input: The user uses a predefined model instance by specifying a configuration. In addition, the user indicates the compilation process using the first compiler, the second compiler, and the third compiler through compilation parameters.

[0132] Compile phase:

[0133] Torch compilation encapsulation: The model instance submitted by the user is encapsulated through the torch.compile API, which serves as a front-end interface to trigger the subsequent compilation process.

[0134] The first compiler performs bytecode analysis: the first compiler receives Python bytecode (ie, the Python language implementation of the model instance), and parses a Python code object (PyCodeObject) from a Python stack frame object (PythonFrameObject) of the Python runtime environment.

[0135] The first compiler builds a functional expression graph (FX graph) based on the bytecode in the parsed PyCodeObject, and inserts a guard mechanism in the process to support dynamic code optimization. At the same time, the code is preliminarily compiled.

[0136] The compiled code replaces the original Python bytecode and generates a converted PyCodeObject.

[0137] The second compiler uses the __torch_dispatch__ mechanism to track the forward and backward propagation processes and generate a joint computational graph containing forward and backward propagation.

[0138] The second compiler further converts the joint computation graph to the Aten / Prim operator level, and then the dispatcher distributes the operator to the appropriate kernel function for execution based on the properties of the input tensor processing module.

[0139] The third compiler receives the graph generated by the second compiler and gradually converts the operator to the intermediate representation (Triton IR). Then, Triton IR is converted to a multi-layer intermediate representation (MLIR). The multi-layer intermediate representation framework performs deep optimization on the computational graph, including operator fusion, constant folding, traversal optimization, etc. Finally, MLIR is converted to LLVM IR by the operator and generates a deeply optimized kernel function.

[0140] Model deployment and training phase:

[0141] After the third compiler completes all conversions and optimizations, it generates and returns a static graph to the large language model training framework. The operations in the static graph directly call the deeply optimized kernel function.

[0142] The large language model training framework uses PyTorch's Distributed Data Parallel (DDP) interface to encapsulate the compiled static graph and start the first round of data training.

[0143] When processing subsequent batches of data, the guard mechanism determines whether recompilation is required. If recompilation is not required, the compiled code is directly read from the cache; otherwise, the recompilation process is triggered.

[0144] The training data is forward propagated through the forward static graph of the static graph to obtain the prediction result, and then the loss value loss is calculated based on the prediction result and the label of the training data. Based on the loss value, the gradient value of each parameter is calculated through the reverse static graph of the static graph using the calculus chain rule.

[0145] After the gradient value is input into the optimizer, the model parameters are updated. Then, the next round of data iteration is entered, and the steps of forward propagation, back propagation and parameter update are repeated to realize the training of the model instance.

[0146] Compared with the traditional dynamic graph mode, the implementation method provided by the present disclosure improves the computing efficiency by about 15%.

[0147] In a possible implementation, when the compilation backend indicated by the compilation parameters includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform compilation operations; the fourth compiler converts operators adapted to the underlying kernel into operators adapted to the specific hardware platform; wherein, when a collective communication operator for integrating processing results of multiple processors is identified, the function of the collective communication operator in the back-propagation computation graph is converted to: obtaining the collective communication results saved in the forward propagation stage from the memory; and performing deep-level optimization on the static graph adapted to the hardware to obtain a deeply optimized static graph.

[0148] In the development and deployment of deep learning models, different hardware platforms (such as GPUs, etc.) may require different compilation strategies to optimize hardware performance. In this implementation, when the compilation parameters indicate that the compilation backend includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler will call the fourth compiler for adapting to the hardware platform according to the compilation parameters, so that the compiled model can run efficiently on the hardware.

[0149] Operators are the basic operation units in model instances, and different hardware platforms have different requirements for the implementation of operators. During the compilation process, the fourth compiler can convert operators that are originally adapted to the underlying kernel into operators that are adapted to a specific hardware platform.

[0150] During the deep learning training process, collective communication operators are used to integrate the computational results from different processors (such as GPUs).

[0151] In order to avoid repeated collective communication operations in the back-propagation phase, in this implementation, the collective communication results completed in the forward propagation phase are saved in the memory and directly obtained from the memory during the back-propagation. In addition, the static graph adapted to the hardware is deeply optimized to further improve the computational efficiency.

[0152] When using the fourth compiler for adaptation, the fourth compiler will recognize the use of this operator and optimize it to calculate and save the collective communication results in the forward propagation phase, and directly obtain these results from the memory in the backward propagation phase, avoiding the GPU lockup problem (GPU lockup) that may be caused by executing the collective communication operation again.

[0153] In addition, the fourth compiler will also perform deep optimization on static graphs, such as adjusting the calculation order, merging similar operations, etc., to further improve the operating efficiency of the model.

[0154] In the disclosed embodiment, by selecting and calling the fourth compiler optimized for a specific hardware platform, efficient compilation and deployment of the deep learning model is achieved; and by optimizing the use of the collective communication operator, performance problems (such as GPU lock) that may be caused by repeated execution of collective communication operations in the back propagation phase are avoided. The efficiency and stability of model instance training on a specific hardware platform are improved.

[0155] Compared with the traditional dynamic graph mode, the implementation method provided by the present disclosure improves the computing efficiency by about 10%.

[0156] In a possible implementation, the model training system can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The system can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the system can be implemented by a server.

[0157] In addition, the present disclosure also provides a model training method, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any model training system provided by the present disclosure. The corresponding technical solutions and descriptions can be found in the corresponding records in the method part and will not be repeated here.

[0158] Figure 2A flow chart of a model training method according to an embodiment of the present disclosure is shown, which is applied to the above system, such as Figure 2 As shown, the method includes:

[0159] In step S21, the large language model training framework receives the model instance and compilation parameters input by the user;

[0160] In step S22, the large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting to compile the model instance;

[0161] In step S23, the model static graph optimization compiler receives the input model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters, performs a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework;

[0162] In step S24, the large language model training framework receives the static graph and trains the model instance based on the static graph.

[0163] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0164] In a case where the compilation backend indicated by the compilation parameter includes a first compiler for capturing a forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation;

[0165] The first compiler obtains the bytecode of the model instance;

[0166] The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler recognizes that there are modules in the model instance that do not support compilation into a static graph, the first compiler skips the compilation operation on the modules that do not support compilation into a static graph; when the first compiler recognizes that there are modules in the model instance that the first compiler does not support compilation operations, the first compiler replaces the modules that do not support compilation operations with equivalent modules with equivalent functions, and performs compilation operations on the codes of the equivalent modules.

[0167] In a possible implementation, the module that does not support compilation into a static graph includes a recalculation module, and the graph compilation operation on the model instance includes:

[0168] The first compiler, when identifying the recalculation module of the model instance, skips the compilation operation of the recalculation module;

[0169] In a possible implementation, the module that the first compiler does not support for performing compilation operations includes an in-place array operation module, and the graph compilation operation on the model instance includes:

[0170] When the code of the in-place array operation module for performing the in-place array operation is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and a compile operation is performed on the code of the non-in-place array operation module.

[0171] In a possible implementation, the training the model instance based on the static graph includes:

[0172] The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data;

[0173] The large language model training framework disables recompilation operations when the amount of data in the last batch of training data is less than the amount of data in the previous batch, and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

[0174] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0175] In a case where the compilation backend indicated by the compilation parameter includes a second compiler for operator conversion, the model static graph optimization compiler calls the second compiler to perform a compilation operation;

[0176] The second compiler captures the forward propagation and reverse propagation computation graphs of the model instance and generates a joint computation graph;

[0177] The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the port of the model static graph optimization compiler is used to encapsulate the custom operator into the operation of the model static graph optimization compiler.

[0178] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0179] In a case where the compilation backend indicated by the compilation parameters includes a third compiler for performing deep-level optimization on the computation graph, the model static graph optimization compiler calls the third compiler to perform a compilation operation;

[0180] The third compiler converts the operator adapted to the underlying kernel into an operator adapted to the hardware, and obtains a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped;

[0181] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0182] In a possible implementation, the performing a graph compilation operation on the model instance includes:

[0183] In a case where the compilation backend indicated by the compilation parameters includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform a compilation operation;

[0184] The fourth compiler converts the operator adapted to the underlying kernel into an operator adapted to the specific hardware platform; wherein, when a collective communication operator for integrating processing results of multiple processors is identified, the function of the collective communication operator in the back propagation calculation graph is converted to: obtaining the collective communication result saved in the forward propagation stage from the memory;

[0185] The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

[0186] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0187] In some embodiments, the functions or modules included in the method provided in the embodiments of the present disclosure can be used to implement the system described in the above system embodiment. The specific implementation can refer to the description of the above system embodiment, and for the sake of brevity, it will not be repeated here.

[0188] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above system when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0189] The embodiment of the present disclosure also proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to implement the above system.

[0190] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements the above system.

[0191] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0192] Figure 3 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0193] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0194] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0195] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0196] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0197] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0198] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0199] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0200] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0201] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0202] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0203] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0204] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0205] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0206] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0207] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A model training system, characterized in that: include: Large language model training framework, receiving model instances and compilation parameters input by users; The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting compilation of the model instance; The model static graph optimization compiler receives the incoming model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework; The large language model training framework receives the static graph and trains the model instance based on the static graph.

2. The system according to claim 1, characterized in that In a case where the compilation backend indicated by the compilation parameter includes a first compiler for capturing a forward propagation computation graph, the model static graph optimization compiler calls the first compiler to perform a compilation operation; The first compiler obtains the bytecode of the model instance; The first compiler compiles the bytecode to construct a static graph of the model; wherein, when the first compiler recognizes that there are modules in the model instance that do not support compilation into a static graph, the first compiler skips the compilation operation on the modules that do not support compilation into a static graph; when the first compiler recognizes that there are modules in the model instance that the first compiler does not support compilation operations, the first compiler replaces the modules that do not support compilation operations with equivalent modules with equivalent functions, and performs compilation operations on the codes of the equivalent modules.

3. The system according to claim 2, characterized in that The module that does not support compilation into a static graph includes a recalculation module, and the first compiler, when identifying the recalculation module of the model instance, skips the compilation operation of the recalculation module; The modules that the first compiler does not support performing compilation operations include in-place array operation modules. When the code of the in-place array operation module used to perform in-place array operations is identified, the code of the in-place array operation module is replaced with the code of the non-in-place array operation module, and the compilation operation is performed on the code of the non-in-place array operation module.

4. The system according to claim 1, characterized in that The large language model training framework trains the model instance based on the static graph for each batch of training data with the same amount of data in the training data; The large language model training framework disables recompilation operations when the amount of data in the last batch of training data is less than the amount of data in the previous batch, and trains the model instance based on the dynamic graph of the model instance using the last batch of training data.

5. The system according to claim 1, characterized in that In a case where the compilation backend indicated by the compilation parameter includes a second compiler for operator conversion, the model static graph optimization compiler calls the second compiler to perform a compilation operation; The second compiler captures the forward propagation and reverse propagation computation graphs of the model instance and generates a joint computation graph; The second compiler converts the operators in the joint computation graph into operators adapted to the underlying kernel; wherein, when a custom operator in the large language model training framework is identified, the port of the model static graph optimization compiler is used to encapsulate the custom operator into the operation of the model static graph optimization compiler.

6. The system according to claim 1, characterized in that In a case where the compilation backend indicated by the compilation parameters includes a third compiler for performing deep-level optimization on the computation graph, the model static graph optimization compiler calls the third compiler to perform a compilation operation; The third compiler converts the operator adapted to the underlying kernel into an operator adapted to the hardware, and obtains a static graph adapted to the hardware; wherein, when an operator type that cannot be converted is identified, the operator type that cannot be converted is skipped; The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

7. The system according to claim 6, characterized in that In a case where the compilation backend indicated by the compilation parameters includes a fourth compiler for adapting to a specific hardware platform, the model static graph optimization compiler calls the fourth compiler to perform a compilation operation; The fourth compiler converts the operator adapted to the underlying kernel into an operator adapted to the specific hardware platform; wherein, when a collective communication operator for integrating processing results of multiple processors is identified, the function of the collective communication operator in the back propagation calculation graph is converted to: obtaining the collective communication result saved in the forward propagation stage from the memory; The static graph of the adapted hardware is optimized in depth to obtain a deeply optimized static graph.

8. A model training method, characterized in that: A system as claimed in any one of claims 1 to 7, comprising: Large language model training framework, receiving model instances and compilation parameters input by users; The large language model training framework passes the model instance and compilation parameters to the port of the model static graph optimization compiler, requesting compilation of the model instance; The model static graph optimization compiler receives the incoming model instance and compilation parameters, uses the compilation backend corresponding to the compilation parameters to perform a graph compilation operation on the model instance, obtains a static graph of the model instance, and returns it to the large language model training framework; The large language model training framework receives the static graph and trains the model instance based on the static graph.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the system described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the system according to any one of claims 1 to 7 is implemented.

Images