A model training method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a model training method and device, an electronic device and a storage medium. The method comprises: constructing a Docker image according to a first model training framework, a first software stack corresponding to a first GPU, and preset model training parameters, the first model training framework being a model training framework for calling the first GPU for model training, the first model training framework being obtained by performing parameter conversion on a second model training framework, the second model training framework being a model training framework for calling a second GPU for model training; sending the Docker image to a plurality of training nodes, and controlling each training node to create a Docker container based on the Docker image; and for any one training node, controlling the training node to call the first GPU deployed on the training node to perform model pre-training in the Docker container. The embodiments of the present disclosure can effectively realize rapid deployment and rapid training of model training.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method and apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, large language models have sparked a surge in the field of artificial intelligence, with many companies seeking easier, faster, and more economical ways to train and deploy their own large language models. However, as models grow larger, training costs increase accordingly. Training such large models requires expensive multi-GPU clusters with multiple nodes, resulting in high hardware resource costs. Even with GPU clusters, existing open-source systems suffer from low machine utilization during training. Megatron-DeepSpeed ​​is an open-source deep learning optimization library designed to improve the efficiency and scalability of large-scale model training. Currently, there is a pressing need for a model training method that allows for distributed large-scale model training using a framework like Megatron-DeepSpeed, adapted to existing GPU clusters. Summary of the Invention

[0003] This disclosure presents a technical solution for a model training method and apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a model training method is provided, comprising: constructing a Docker image based on a first model training framework, a first software stack corresponding to a first GPU, and preset model training parameters, wherein the first model training framework is a model training framework that calls the first GPU for model training, the first model training framework is obtained by parameter transformation of a second model training framework, the second model training framework is a model training framework that calls a second GPU for model training, and the first GPU and the second GPU are GPUs with different architecture configurations; sending the Docker image to multiple training nodes, and controlling each training node to create a Docker container based on the Docker image, wherein the first GPU is deployed on each training node; and for any training node, controlling the training node to call the first GPU deployed on the training node for model pre-training within the Docker container.

[0005] In one possible implementation, the method further includes: performing parameter conversion on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework; calling the compiler corresponding to the first GPU to perform code compilation operation on the second source code to obtain the target executable file corresponding to the first model training framework.

[0006] In one possible implementation, the first model training framework includes: a first type of application programming interface (API), a first type of kernel function, a first type of header file, and a first type of communication library adapted to the first software stack; the step of performing parameter conversion on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework includes: performing an interface conversion operation on the second type API in the first source code to obtain the first type API, wherein the second type API is an API adapted to the second software stack corresponding to the second GPU; performing a replacement operation on the second type kernel function in the first source code to obtain the first type kernel function, wherein the second type kernel function is a kernel function adapted to the second software stack; performing a replacement operation on the second type header file in the first source code to obtain the first type header file, wherein the second type header file is a header file adapted to the second software stack; and performing a replacement operation on the second type communication library in the first source code to obtain the first type communication library, wherein the second type communication library is a communication library adapted to the second software stack.

[0007] In one possible implementation, the method further includes: filtering a first type of operator in the first source code, and setting comment tags for the first type of operator, wherein the first type of operator is an operator incompatible with the first software stack, and the comment tags are used to indicate that the first type of operator is not run.

[0008] In one possible implementation, the method further includes: performing data type adaptation processing on the first software stack according to the multiple data types supported by the mixed precision acceleration model in the first model training framework, to obtain an updated first software stack, wherein the updated first software stack supports the multiple data types.

[0009] According to one aspect of this disclosure, a model training apparatus is provided, comprising: a Docker image building module, configured to build a Docker image based on a first model training framework, a first software stack corresponding to a first GPU, and preset model training parameters, wherein the first model training framework is a model training framework that calls the first GPU for model training, the first model training framework is obtained by parameter transformation of a second model training framework, the second model training framework is a model training framework that calls a second GPU for model training, and the first GPU and the second GPU are GPUs with different architecture configurations; a Docker container creation module, configured to send the Docker image to multiple training nodes and control each training node to create a Docker container based on the Docker image, wherein the first GPU is deployed on each training node; and a pre-training module, configured to control any training node to call the first GPU deployed on the training node for model pre-training within the Docker container.

[0010] In one possible implementation, the apparatus further includes a training framework determination module, specifically configured to: perform parameter conversion on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework; call the compiler corresponding to the first GPU to perform code compilation operation on the second source code to obtain the target executable file corresponding to the first model training framework.

[0011] In one possible implementation, the first model training framework includes: a first type of application programming interface (API), a first type of kernel function, a first type of header file, and a first type of communication library adapted to the first software stack; the training framework determination module is specifically used for: performing an interface conversion operation on the second type of API in the first source code to obtain the first type of API, wherein the second type of API is an API adapted to the second software stack corresponding to the second GPU; performing a replacement operation on the second type of kernel function in the first source code to obtain the first type of kernel function, wherein the second type of kernel function is a kernel function adapted to the second software stack; performing a replacement operation on the second type of header file in the first source code to obtain the first type of header file, wherein the second type of header file is a header file adapted to the second software stack; and performing a replacement operation on the second type of communication library in the first source code to obtain the first type of communication library, wherein the second type of communication library is a communication library adapted to the second software stack.

[0012] In one possible implementation, the training framework determination module is specifically used to: filter a first type of operator in the first source code, and set annotation tags for the first type of operator, wherein the first type of operator is an operator incompatible with the first software stack, and the annotation tags are used to indicate that the first type of operator is not run.

[0013] In one possible implementation, the training framework determination module is specifically used to: perform data type adaptation processing on the first software stack according to the multiple data types supported by the mixed precision acceleration model in the first model training framework, to obtain an updated first software stack, wherein the updated first software stack supports the multiple data types.

[0014] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0015] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0016] In this embodiment, by performing parameter conversion on the second model training framework adapted to the second GPU, a first model training framework adapted to the first GPU is determined. The first GPU and the second GPU are GPUs with different architecture configurations. Then, based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters, a Docker image is built to effectively solidify the model training environment setup steps. After sending the Docker image to multiple training nodes, each training node is controlled to create a Docker container based on the Docker image, so that the Docker container of each training node is configured with the same model training environment. For any training node, the training node is controlled to call the first GPU deployed on the training node for model pre-training within the Docker container, effectively realizing rapid deployment and rapid training of model training.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0019] Figure 1 A flowchart of a model training method according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0023] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0024] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0025] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0026] Figure 1 A flowchart illustrating a model training method according to an embodiment of this disclosure is shown. This method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. This method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server. Figure 1 As shown, the method includes:

[0027] In step S11, a Docker image is constructed based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters. The first model training framework is a model training framework that calls the first GPU to train the model. The first model training framework is obtained by converting the parameters of the second model training framework. The second model training framework is a model training framework that calls the second GPU to train the model. The first GPU and the second GPU are GPUs with different architecture configurations.

[0028] The model training framework in this embodiment can be a Megatron-DeepSpeed ​​framework that combines Megatron-LM and DeepSpeed ​​technologies. This framework utilizes techniques such as model parallelism, data parallelism, and pipeline parallelism to address the memory and computational challenges encountered when training large-scale models (e.g., large language models). The first model training framework is a first Megatron-DeepSpeed ​​framework adapted to a first GPU, and the second model training framework is a second Megatron-DeepSpeed ​​framework adapted to a second GPU. The first GPU and the second GPU can be GPUs from different GPU manufacturers; that is, the first GPU and the second GPU are GPUs with different architecture configurations.

[0029] The second model training framework can be a model training framework that is already maturely used in related technologies. Based on the second model training framework, in order to make the model training framework adaptable to the new GPU: the first GPU, the first model training framework can be quickly obtained by transforming the parameters of the second model training framework.

[0030] The following sections will describe in detail the specific process of determining the first model training framework, taking into account the possible implementation methods of this disclosure; these details will not be elaborated upon here.

[0031] Since configuring the training environment is very complex, and in a distributed training scenario, the same training environment needs to be configured on each training node, in order to simplify the configuration process, the steps for setting up the training environment can be solidified into a Docker file based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters, and then a Docker image can be built based on the Docker file.

[0032] In step S12, the Docker image is sent to multiple training nodes, and each training node is controlled to create a Docker container based on the Docker image, wherein a first GPU is deployed on each training node.

[0033] The Docker image constructed above is sent to multiple training nodes, and each training node is controlled to create a Docker container based on the Docker image, thereby effectively ensuring that the Docker containers deployed on each training node have the same training environment.

[0034] In step S13, for any training node, control the training node to call the first GPU deployed on the training node for model pre-training within the Docker container.

[0035] After deploying Docker containers with the same training environment on each training node, distributed training can be started through the master node. This allows each training node to call the first GPU deployed on that training node for model pre-training within the Docker containers deployed on that node.

[0036] The large-scale models targeted for model pre-training can be flexibly set according to the actual situation, such as large language models. This disclosure does not make specific limitations on this.

[0037] In this embodiment, by performing parameter conversion on the second model training framework adapted to the second GPU, a first model training framework adapted to the first GPU is determined. Then, based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters, a Docker image is constructed to effectively solidify the model training environment setup steps. After sending the Docker image to multiple training nodes, each training node is controlled to create a Docker container based on the Docker image, so that the Docker container of each training node is configured with the same model training environment. For any training node, the training node is controlled to call the first GPU deployed on the training node for model pre-training within the Docker container, effectively realizing rapid deployment and rapid training of the model.

[0038] In one possible implementation, the method further includes: performing parameter conversion on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework; calling the compiler corresponding to the first GPU to perform code compilation operation on the second source code to obtain the target executable file corresponding to the first model training framework.

[0039] Since the second model training framework is adapted to the second GPU, and the working parameters of the second GPU and the first GPU may be different, the parameters of the first source code corresponding to the second model training framework are converted. The parameters in the first source code adapted to the second GPU are converted to the parameters adapted to the first GPU. The second source code obtained after the conversion is the source code corresponding to the first model training framework.

[0040] In one example, a parameter conversion tool can be used to filter and convert relevant parameters in the first source code. The specific form of the parameter conversion tool can be flexibly set according to the actual situation, and this disclosure does not impose any specific limitations on it.

[0041] After performing parameter conversion at the source code level, in order to ensure that the second source code can run smoothly, the compiler corresponding to the first GPU is called to perform code compilation operation on the second source code to obtain the target executable file corresponding to the first model training framework.

[0042] Source code is a text file written in a programming language, which includes the program's logic and instructions; executable file is a binary file compiled from the source code, which includes instructions that the computer can directly execute.

[0043] When the second model training framework is the second Megatron-DeepSpeed ​​framework, which is developed based on the PyTorch framework, the first software stack corresponding to the first GPU includes the torch_aaa component. This component is a PyTorch plugin that allows PyTorch to perform model training and inference on the first GPU, where aaa is the identifier of the first software stack. Therefore, the first software stack can be registered as a custom backend in the second Megatron-DeepSpeed ​​framework. Then, parameters in the first source code corresponding to the second Megatron-DeepSpeed ​​framework that are adapted to the second software stack corresponding to the second GPU can be transformed to obtain parameters adapted to the first software stack. This allows calls from the second Megatron-DeepSpeed ​​framework to the second software stack to be converted into calls from the first Megatron-DeepSpeed ​​framework to the first software stack.

[0044] In one possible implementation, the first model training framework includes: a first type of application programming interface (API), a first type of kernel function, a first type of header file, and a first type of communication library adapted to the first software stack; parameter conversion is performed on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework, including: performing an interface conversion operation on the second type API in the first source code to obtain the first type API, wherein the second type API is an API adapted to the second software stack corresponding to the second GPU; performing a replacement operation on the second type kernel function in the first source code to obtain the first type kernel function, wherein the second type kernel function is a kernel function adapted to the second software stack; performing a replacement operation on the second type header file in the first source code to obtain the first type header file, wherein the second type header file is a header file adapted to the second software stack; and performing a replacement operation on the second type communication library in the first source code to obtain the first type communication library, wherein the second type communication library is a communication library adapted to the second software stack.

[0045] By performing parameter conversion on the second type of API, second type of kernel function, second type of header file, and second type of communication library in the first source code corresponding to the second model training framework, the second source code corresponding to the first model training framework adapted to the first GPU can be obtained.

[0046] In the second model training framework, there is usually a lot of interaction with the second GPU backend. Therefore, in the first source code corresponding to the second model training framework, the second software stack corresponding to the second GPU is used as the default backend. That is, the second type API in the first source code is the API adapted to the second software stack. Therefore, in the first source code, an interface conversion operation is performed on the second type API to obtain the first type API adapted to the first software stack corresponding to the first GPU.

[0047] For example, in the device configuration, the first source code sets the second software stack as the GPU backend using the interface parameter `pretrained("gpt2").to("bbb")`, where `bbb` is the identifier of the second software stack. After performing an interface conversion operation on the first source code, the second source code is obtained. In the second source code, the first software stack is set as the GPU backend using the interface parameter `pretrained("gpt2").to("aaa")`, where `aaa` is the identifier of the first software stack.

[0048] In one example, API adaptation may include not only setting the device as described above, but also setting the synchronization interface, setting the stream, modifying the data type, determining whether the backend is available, modifying the C++ code interface, etc. It may also include other API adaptations as needed, and this disclosure does not make any specific limitations on this.

[0049] The first source code corresponding to the second model training framework includes many second-type kernel functions adapted to the second software stack. Since the first software stack has good compatibility with the second software stack, mainly reflected in the similarity between the interfaces of the driver functions in the first and second software stacks, and their functionalities are also largely the same, the interfaces of the second-type kernel functions adapted to the second software stack in the first source code can be quickly replaced using the parameter conversion tool corresponding to the first software stack, thus obtaining the first-type kernel functions adapted to the first software stack in the second source code.

[0050] The first source code corresponding to the second model training framework usually includes many second-type header files adapted to the second software stack (e.g., some C++ header files, header files corresponding to the second-type kernel functions mentioned above, etc.). Therefore, in order to adapt to the first software stack, it is necessary to perform a replacement operation on the second-type header files to obtain the first-type header files adapted to the first software stack.

[0051] In the second model training framework, a second type of communication library adapted to the second software stack is used as the distributed communication library of the second GPU backend. In order to adapt to the first software stack, the second type of communication library in the second model training framework needs to be replaced to obtain the first type of communication library, so that the first model training framework can use the first type of communication library adapted to the first software stack as the distributed communication library of the first GPU backend.

[0052] In one possible implementation, the method further includes: filtering a first type of operator in the first source code, and setting comment labels for the first type of operator, wherein the first type of operator is an operator incompatible with the first software stack, and the comment labels are used to indicate that the first type of operator is not run.

[0053] In the first source code corresponding to the second model training framework, by setting comment labels for the first type of operators that are incompatible with the first software stack, it is indicated that these operators will not be run in the first model training framework.

[0054] For example, if the first software stack does not support the compilation capability of torch.jit, operators such as bias_gelu and bias_dropout_add compiled by torch.jit.script in the second model training framework are selected and identified as first-type operators, and annotation labels are set for these first-type operators.

[0055] For the first type of operator with annotation labels, since the native PyTorch operator is adapted to the first software stack, the first model training framework can use the native PyTorch operator with the same functionality to replace the first type of operator.

[0056] In addition to the examples mentioned above, the first type of operator may include other operators depending on the actual situation, and this disclosure does not make specific limitations on this.

[0057] In one possible implementation, the method further includes: performing data type adaptation processing on the first software stack according to the multiple data types supported by the mixed precision acceleration model in the first model training framework, to obtain an updated first software stack, wherein the updated first software stack supports the multiple data types.

[0058] When a mixed-precision acceleration model is used in the first model training framework to speed up model training, the first software stack may not be able to support the various data types supported by the mixed-precision acceleration model. In order to successfully run the first model training framework on the first software stack, it is necessary to perform data type adaptation processing on the first software stack according to the various data types supported by the mixed-precision acceleration model in the first model training framework, so as to obtain an updated first software stack that can support the various data types required by the mixed-precision acceleration model.

[0059] When the first model training framework is the first Megatron-DeepSpeed ​​framework, the mixed-precision accelerated model training used in the first Megatron-DeepSpeed ​​framework leverages the advantages of half-precision floating-point (e.g., FP16 or BF16) data types—lower memory footprint and faster computation—for matrix multiplication operations. Simultaneously, it utilizes the wider representation range of Double and FP32 data types to address data overflow and rounding errors. However, if the first software stack does not support the BF16 data type, then to adapt the first software stack to the half-precision floating-point data type of the mixed-precision accelerated model in the first model training framework, data type adaptation processing is required. This ensures that the updated first software stack supports the FP16 data type and also supports the Double type.

[0060] In one example, the large model to be trained includes embedding operators, but if the target operator library (e.g., the DNN operator library) included in the first software stack does not support the FP16 data type of the embedding operators, then data type adaptation processing needs to be performed on the embedding operators in the target operator library in the first software stack so that the updated embedding operators in the target operator library in the first software stack support the FP16 data type.

[0061] For example, data type adaptation processing is performed in the forward propagation kernel function and backward propagation kernel function of the Embedding operator in the target operator library in the first software stack to enable it to support the FP16 data type.

[0062] In one example, an operator scan can be performed on the large model to be trained to determine if there are any target operators in the large model that are not adapted to the FP16 data type in the torch_aaa component included in the first software stack. For example, after the operator scan, it is determined that target operators such as Embedding, LayerNorm, Baddbmm, and Dropout included in the large model are not adapted to the FP16 data type in the torch_aaa component. At this time, data type adaptation processing needs to be performed on the first software stack so that the torch_aaa component included in the first software stack supports the FP16 data type of the target operators.

[0063] The adaptation process may include: for the target operator in the first software stack, locating the code segment in the `torch_aaa` component that performs data type checks on the target operator, adapting the data type of this code segment, and updating the code segment to include checks on the FP16 data type. By extending the data type checks, the updated `torch_aaa` component can support the FP16 data type of the target operator. Since the target operator library (e.g., the DNN operator library) already supports the FP16 data type, when the updated `torch_aaa` component calls a kernel function of the FP16 data type, it can call the FP16 data type implementation in the target operator library.

[0064] To address the adaptation of the torch_aaa component mentioned above, a git patch can be generated so that when compiling the unupdated torch_aaa component, this git patch will be compiled simultaneously, thereby applying the above adaptation to the torch_aaa component and obtaining the updated torch_aaa component.

[0065] In one example, the second GPU typically includes Tensor Core computation units, making the computational power of the BF16 data type significantly higher than that of FP32 and FP16 data types. Therefore, the fusion kernel function of the second model training framework also uses the BF16 data type to accelerate computation. However, if the first GPU does not include Tensor Core units, meaning the first software stack does not support the BF16 data type, then the fusion kernel function of the second model training framework needs to be adapted to use the FP16 data type instead of the BF16 data type.

[0066] In one example, to prevent floating-point data type overflow, the first model training framework uses the Double data type for some parameters. These parameters require the reduce kernel function during the summation process. Therefore, in the first software stack, the reduce kernel function needs to be adapted to support the Double data type so that the updated reduce kernel function supports the Double data type.

[0067] After completing the parameter conversion and data type adaptation processes at the code level, the first model training framework can be applied to the first GPU to perform model test training. During the test training process, any abnormal issues are detected. For any abnormal issues, further adaptation processing is performed on the first model training framework and / or the first software stack according to the type of issue, ultimately enabling the first model training framework to be successfully applied to the first GPU to execute the pre-training of a large model.

[0068] In one example, the issue type is: the torch_aaa component in the first software stack uses the Gemm kernel function implemented in the aaa_BLAS library by default, but the Gemm kernel function does not support the FP16 data type. Based on the issue type, adaptation processing is performed on the first software stack: since the aaa_DNN library in the first software stack supports the Gemm kernel function for the FP16 data type, the interface calling the Gemm kernel function in the aaa_BLAS library in the torch_aaa component is converted to call the Gemm kernel function in the aaa_DNN library.

[0069] In one example, the anomaly is a training loss of NaN (Not a Number). The first model training framework uses automatic mixed precision training. During training, the Scaler parameter is used to amplify / scale the training loss to minimize gradient underflow. Since the reciprocal function in the aaa_DNN library of the first software stack does not support the Double data type, the Scaler parameter keeps increasing with the number of training steps. When this parameter grows to a certain point, 1 / Scaler will underflow, resulting in a NaN training loss. Therefore, an adaptation process is performed on the first software stack based on the problem type: the reciprocal function in the aaa_DNN library is adapted to support the Double data type in the updated form.

[0070] In one example, the anomaly type is operator out of bounds. In large language models, matrix operations are typically very large, and distributed training further increases the batch size of the input data. When the batch size reaches a certain number (e.g., 64), the Index operator will go out of bounds, leading to training crashes. Based on the problem type, the first software stack is adapted: according to the training process requirements, the matrix index INTM data type is determined (where M is a positive integer, representing the maximum batch size that can be supported under successful training conditions). Then, the Index kernel function in the aaa_DNN library of the first software stack is adapted to support the INTM data type so that the updated Index kernel function supports the INTM data type.

[0071] In addition to the examples mentioned above, the types of problems and corresponding adaptation methods may include other content depending on the actual situation, and this disclosure does not make specific limitations on this.

[0072] After completing all the above adaptation processes for the first model training framework and the first software stack, a Docker image can be built based on the first model training framework, the first software stack, and the preset model training parameters corresponding to the large model to be trained, so as to solidify the training environment configuration steps. The Docker image built above is sent to multiple training nodes, and each training node is controlled to create a Docker container based on the Docker image, thereby effectively ensuring that the Docker containers deployed on each training node have the same training environment. After the Docker containers with the same training environment are deployed on each training node, distributed training can be started through the master node, and each training node can be controlled to call the first GPU deployed on the training node for model pre-training within the Docker containers deployed on that node.

[0073] In this embodiment, by performing parameter conversion on the second model training framework adapted to the second GPU, a first model training framework adapted to the first GPU is determined. The first GPU and the second GPU are GPUs with different architecture configurations. Then, based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters, a Docker image is built to effectively solidify the model training environment setup steps. After sending the Docker image to multiple training nodes, each training node is controlled to create a Docker container based on the Docker image, so that the Docker container of each training node is configured with the same model training environment. For any training node, the training node is controlled to call the first GPU deployed on the training node for model pre-training within the Docker container, effectively realizing rapid deployment and rapid training of model training.

[0074] When using the first GPU to perform distributed pre-training of a large model based on the first model training framework, there may be significant performance loss. End-to-end pre-training requires a large amount of resources and is very time-consuming, resulting in very high pre-training costs. Therefore, this disclosure further analyzes the performance issues of the first software stack during distributed pre-training of a large model using the first GPU based on the first model training framework, in order to reduce the resources required for distributed pre-training of large models and reduce the time consumption.

[0075] In one example, performance analysis is performed at the component level (torch) of the model training framework.

[0076] First, based on the first model training framework, the first GPU is called to perform large-scale distributed training of the model, and the first performance data is captured during the large-scale distributed training process.

[0077] Then, based on the second model training framework, the second GPU is called to perform large-scale distributed training of the model, and second performance data is captured during the large-scale distributed training process.

[0078] Finally, by analyzing the first and second performance data, the first model training framework and the first software stack are improved.

[0079] In one example, by analyzing the first performance data, the first time consumption of multiple target operators during the process of the first model training framework calling the first GPU for large-scale distributed training is determined; by analyzing the second performance data, the second time consumption of multiple target operators during the process of the second model training framework calling the second GPU for large-scale distributed training is determined; by comparing the first time consumption and the second time consumption, the first software stack is improved.

[0080] In one example, by comparing the first and second execution times, it can be determined that the second execution time is higher than the first execution time due to the mismatch of the first software stack operator.

[0081] By comparing the first and second execution times, it was determined that operators such as masked_fill, index_put, and layer_norm were not adapted in the first software stack, resulting in unreasonable data copying of these operators on the first GPU, which caused the first execution time of these operators to be higher than the second execution time.

[0082] The following analysis uses the index_put operator as an example to illustrate how to improve the first software; other operators are similar.

[0083] In the second GPU, the index_put operator is invoked during index assignment.

[0084] However, in the first GPU, because the first software stack is not adapted to the `index_put` operator, the `index_put` operator copies the data to the CPU, then calls the PyTorch implementation on the CPU, and finally copies it back to the first GPU after execution. This results in two data copies, causing higher processing time.

[0085] Therefore, the index_put operator is adapted in the first software stack so that in the first GPU, the index_put operator is called when the index is assigned, and there is only one data copy time.

[0086] In one example, by comparing the first and second timeouts, it can be determined that there are multiple consecutive data copies in the forward process, causing the second timeout to be longer than the first timeout.

[0087] The first data copy is not far from the `lt` operator on the left, and the corresponding code is identified as the index assignment part in the `build_mask_matrix` function. Multiple clones exist afterward. The last clone's subsequent operators include `ge`, `is_nonzero`, `lt`, and `sub`, indicating these operators have entered the Vocabulary Parallel Embedding. Therefore, this part likely converts the model input from float32 to float16, with the corresponding code located in the GPT Model Pipe. This part involves numerous small operator data copies, resulting in low computational density and memory utilization. Consequently, multiple consecutive data copies during the forward process lead to high latency.

[0088] Therefore, the first software stack is optimized by merging this part of the code into a larger fusion operator, which improves memory access efficiency, improves overall performance, and thus effectively reduces time consumption.

[0089] In one example, by comparing the first and second execution times, it can be determined that there are many small operators in the forward process that cause the second execution time to be higher than the first execution time.

[0090] Analysis revealed that the forward process in the second software stack contains numerous short operators. Since these operators precede the `build_single_mask` location, they primarily occur during data preprocessing. These operators are then called in a cyclical manner using the `select-gt-select-is_nonzero-mul` method. The determination location is `_build_key_size_numel_dictionaries`. Code analysis showed that this continuous operation runs on the CPU, taking approximately 1ms in total.

[0091] Therefore, by optimizing the first software stack and running these small operator operations uniformly on the first GPU, the time consumption was reduced from 1ms to 0.17ms, which effectively reduced the time consumption.

[0092] In one example, performance analysis is performed at the kernel function level of the model training framework.

[0093] First, based on the first model training framework, the first GPU is used for distributed training of a large model, and the roofline curve is determined during the distributed training process. The roofline curve is used to evaluate the relationship between computational performance and hardware resource utilization.

[0094] Then, based on the roofline curve, throughput is calculated to optimize computationally intensive kernel functions, and memory access efficiency is calculated to optimize memory-intensive kernel functions.

[0095] In one example, based on the throughput calculated from the roofline curve, it is determined that there is a computationally intensive kernel function whose computational efficiency is less than expected at certain sizes. In this case, performance optimization can be performed on the computationally intensive kernel function at these sizes.

[0096] In one example, the memory access efficiency calculated from the roofline curve indicates that there exists a memory-intensive kernel function (e.g., unary kernel function, gelu, etc.) whose memory access efficiency is lower than expected. This is because the input data of these kernel functions involves a transpose operation, which causes the non-continuous torch to call the contiguous function to make the input data continuous before calling the DNN operator for computation. However, the unary kernel function is computed element-wise and does not require the input data to be made continuous.

[0097] Therefore, the Contiguous function can be removed, and when creating the Tensor, CreateMUTensor, which supports stride propagation, can be used to pass the stride attribute of the input data. Then, the unary kernel function of the DNN operator can be directly called for computation, thereby making the memory access efficiency meet expectations.

[0098] Regarding the performance issues of the first software stack during the large model pre-training process using the first GPU based on the first model training framework, further analysis can be conducted from other aspects based on actual needs, and this disclosure does not impose specific limitations on this.

[0099] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0100] In addition, this disclosure also provides a model training device, electronic device, computer-readable storage medium, and program, all of which can be used to implement any of the model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0101] Figure 2 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown. Figure 2 As shown, device 20 includes:

[0102] Docker image building module 21 is used to build a Docker image based on the first model training framework, the first software stack corresponding to the first GPU, and preset model training parameters. The first model training framework is a model training framework that calls the first GPU to train the model. The first model training framework is obtained by converting the parameters of the second model training framework. The second model training framework is a model training framework that calls the second GPU to train the model. The first GPU and the second GPU are GPUs with different architecture configurations.

[0103] Docker container creation module 22 is used to send Docker images to multiple training nodes and control each training node to create Docker containers based on the Docker images, wherein each training node has a first GPU deployed on it;

[0104] The pre-training module 23 is used to control any training node to call the first GPU deployed on the training node for model pre-training within the Docker container.

[0105] In one possible implementation, device 20 further includes: a training framework determination module, specifically used for:

[0106] The parameters of the first source code corresponding to the second model training framework are transformed to obtain the second source code corresponding to the first model training framework.

[0107] The compiler corresponding to the first GPU is invoked to perform code compilation operations on the second source code, resulting in the target executable file corresponding to the first model training framework.

[0108] In one possible implementation, the first model training framework includes: a first type of application programming interface (API) adapted to the first software stack, a first type of kernel function, a first type of header file, and a first type of communication library;

[0109] The training framework determination module is specifically used for:

[0110] In the first source code, an interface conversion operation is performed on the second type API to obtain the first type API, wherein the second type API is an API adapted to the second software stack corresponding to the second GPU;

[0111] In the first source code, a replacement operation is performed on the second type kernel function to obtain the first type kernel function, wherein the second type kernel function is a kernel function adapted to the second software stack;

[0112] In the first source code, a replacement operation is performed on the second type header file to obtain the first type header file, wherein the second type header file is a header file adapted to the second software stack;

[0113] In the first source code, a replacement operation is performed on the second type of communication library to obtain the first type of communication library, wherein the second type of communication library is a communication library adapted to the second software stack.

[0114] In one possible implementation, the training framework determines a module, specifically for:

[0115] In the first source code, a first type of operator is filtered, and a comment label is set for the first type of operator, wherein the first type of operator is an operator that is incompatible with the first software stack, and the comment label is used to indicate that the first type of operator is not run.

[0116] In one possible implementation, the training framework determines a module, specifically for:

[0117] Based on the multiple data types supported by the mixed-precision acceleration model in the first model training framework, data type adaptation processing is performed on the first software stack to obtain the updated first software stack, wherein the updated first software stack supports multiple data types.

[0118] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0119] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0120] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0121] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0122] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0123] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0124] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 3 The electronic device 1900 can be provided as a server or a terminal device. (See reference...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0125] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0126] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0127] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0131] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0136] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0137] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0138] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0139] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that, include: A Docker image is constructed based on the first model training framework, the first software stack corresponding to the first GPU, and the preset model training parameters. The first model training framework is a model training framework that calls the first GPU to train the model. The first model training framework is obtained by converting the parameters of the second model training framework. The second model training framework is a model training framework that calls the second GPU to train the model. The first GPU and the second GPU are GPUs with different architecture configurations. The Docker image is sent to multiple training nodes, and each training node is controlled to create a Docker container based on the Docker image, wherein the first GPU is deployed on each training node; For any training node, control that training node to call the first GPU deployed on the training node within the Docker container to perform model pre-training.

2. The method according to claim 1, characterized in that, The method further includes: The parameters of the first source code corresponding to the second model training framework are transformed to obtain the second source code corresponding to the first model training framework. The compiler corresponding to the first GPU is invoked to perform code compilation operations on the second source code, thereby obtaining the target executable file corresponding to the first model training framework.

3. The method according to claim 2, characterized in that, The first model training framework includes: a first type of application programming interface (API), a first type of kernel function, a first type of header file, and a first type of communication library adapted to the first software stack; The step of performing parameter conversion on the first source code corresponding to the second model training framework to obtain the second source code corresponding to the first model training framework includes: In the first source code, an interface conversion operation is performed on the second type API to obtain the first type API, wherein the second type API is an API adapted to the second software stack corresponding to the second GPU; In the first source code, a replacement operation is performed on the second type of kernel function to obtain the first type of kernel function, wherein the second type of kernel function is a kernel function adapted to the second software stack; In the first source code, a replacement operation is performed on the second type of header file to obtain the first type of header file, wherein the second type of header file is a header file adapted to the second software stack; In the first source code, a replacement operation is performed on the second type of communication library to obtain the first type of communication library, wherein the second type of communication library is a communication library adapted to the second software stack.

4. The method according to claim 3, characterized in that, The method further includes: In the first source code, a first type of operator is filtered, and a comment label is set for the first type of operator, wherein the first type of operator is an operator that is incompatible with the first software stack, and the comment label is used to indicate that the first type of operator is not run.

5. The method according to claim 1, characterized in that, The method further includes: Based on the multiple data types supported by the mixed precision acceleration model in the first model training framework, data type adaptation processing is performed on the first software stack to obtain an updated first software stack, wherein the updated first software stack supports the multiple data types.

6. A model training device, characterized in that, include: The Docker image building module is used to build a Docker image based on the first model training framework, the first software stack corresponding to the first GPU, and preset model training parameters. The first model training framework is a model training framework that calls the first GPU to train the model. The first model training framework is obtained by transforming the parameters of the second model training framework. The second model training framework is a model training framework that calls the second GPU to train the model. The first GPU and the second GPU are GPUs with different architecture configurations. A Docker container creation module is used to send the Docker image to multiple training nodes and control each training node to create a Docker container based on the Docker image, wherein the first GPU is deployed on each training node; The pre-training module is used to control any training node to call the first GPU deployed on the training node for model pre-training within the Docker container.

7. The apparatus according to claim 6, characterized in that, The device further includes: a training framework determination module, specifically used for: The parameters of the first source code corresponding to the second model training framework are transformed to obtain the second source code corresponding to the first model training framework. The compiler corresponding to the first GPU is invoked to perform code compilation operations on the second source code, thereby obtaining the target executable file corresponding to the first model training framework.

8. The apparatus according to claim 7, characterized in that, The first model training framework includes: a first type of application programming interface (API), a first type of kernel function, a first type of header file, and a first type of communication library adapted to the first software stack; The training framework determination module is specifically used for: In the first source code, an interface conversion operation is performed on the second type API to obtain the first type API, wherein the second type API is an API adapted to the second software stack corresponding to the second GPU; In the first source code, a replacement operation is performed on the second type of kernel function to obtain the first type of kernel function, wherein the second type of kernel function is a kernel function adapted to the second software stack; In the first source code, a replacement operation is performed on the second type of header file to obtain the first type of header file, wherein the second type of header file is a header file adapted to the second software stack; In the first source code, a replacement operation is performed on the second type of communication library to obtain the first type of communication library, wherein the second type of communication library is a communication library adapted to the second software stack.

9. The apparatus according to claim 8, characterized in that, The training framework determination module is specifically used for: In the first source code, a first type of operator is filtered, and a comment label is set for the first type of operator, wherein the first type of operator is an operator that is incompatible with the first software stack, and the comment label is used to indicate that the first type of operator is not run.

10. The apparatus according to claim 7, characterized in that, The training framework determination module is specifically used for: Based on the multiple data types supported by the mixed precision acceleration model in the first model training framework, data type adaptation processing is performed on the first software stack to obtain an updated first software stack, wherein the updated first software stack supports the multiple data types.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images