Model training method and device and computing device cluster

Abstract

The model training method comprises the steps that a to-be-trained first model is configured at an equipment end, a computational graph of the first model is segmented into N sub computational graphs, and N is larger than or equal to 2; an optimizer of the first model is configured at a host end, the optimizer is segmented to obtain N sub optimizers, and one sub optimizer is responsible for independently executing updating of parameters in one sub calculation graph; the first model is trained through cooperation of the host end and the equipment end, and in the training process, the at least one sub-calculation graph and the at least one sub-optimizer are executed in parallel in the backward propagation calculation. Therefore, by configuring the model at the equipment end, segmenting the calculation graph of the model into a plurality of parts, configuring the optimizer at the host end, segmenting the optimizer into a plurality of parts, and independently executing the updating of the parameters in one sub calculation graph by one sub optimizer, the calculation of the optimizer and the backward propagation calculation can be parallel, and the model training efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method, apparatus, and computing device cluster. Background Technology

[0002] With the advancement of big data and computing power, the complexity of deep learning models continues to rise, driving the rapid development of AI technology. However, this trend has also brought enormous challenges to the model training process, particularly in terms of computing resources and memory management.

[0003] To achieve large-scale model training, the industry has widely adopted various parallelization techniques, including model parallelism, pipeline parallelism, data parallelism, and tensor parallelism. These methods effectively alleviate the problem of insufficient memory on a single computing device by distributing different parts or stages of the model across different computing units, making larger-scale model training possible. However, in practical applications, the efficiency of these methods still needs improvement. Summary of the Invention

[0004] This application provides a model training method, apparatus, computing device cluster, computer storage medium, and computer product that can improve model training efficiency.

[0005] In a first aspect, this application provides a model training method, comprising: configuring a first model to be trained on a device, wherein the computation graph of the first model is divided into N sub-computation graphs, N≥2; configuring an optimizer for the first model on a host, and dividing the optimizer to obtain N sub-optimizers, wherein one sub-optimizer is responsible for independently executing the parameter update of one sub-computation graph; and training the first model through collaboration between the host and the device, wherein during the training process, at least one sub-computation graph is executed in parallel with at least one sub-optimizer in the backpropagation computation.

[0006] In this way, by configuring the model on the Device side and dividing the model's computation graph into multiple parts, and configuring the optimizer on the Host side and dividing the optimizer into multiple parts, with each sub-optimizer responsible for independently updating the parameters in a sub-computation graph, the optimizer computation and backpropagation computation can be parallelized, thereby improving the model training efficiency.

[0007] In one possible implementation, the method further includes: employing mixed-precision training during the training of the first model, and performing a loss scaling overflow check after the optimizer on the host side has finished executing. In this way, mixed-precision training can improve model training efficiency, and by advancing the optimizer's execution order to before the loss scaling overflow check, backpropagation computation and the optimizer can be executed in parallel during mixed-precision training.

[0008] In one possible implementation, the N sub-computation graphs are obtained by splitting the computation graph of the first model at the host end.

[0009] In one possible implementation, the computation graph of the first model and the splitting rules of the optimizer are the same.

[0010] Secondly, this application provides a model training apparatus, comprising: a configuration module and a training module. The configuration module is used to configure a first model to be trained on a device side, wherein the computation graph of the first model is divided into N sub-computation graphs, N≥2; and to configure an optimizer for the first model on a host side, and to divide the optimizer into N sub-optimizers, wherein each sub-optimizer is responsible for independently updating parameters in one sub-computation graph. The training module is used to train the first model collaboratively on the host side and the device side, wherein during training, at least one sub-computation graph is executed in parallel with at least one sub-optimizer in the backpropagation computation.

[0011] In one possible implementation, the training module is also used to: perform mixed-precision training during the training of the first model, and to perform loss scaling overflow judgment after the optimizer on the host has finished executing.

[0012] In one possible implementation, the N sub-computation graphs are obtained by splitting the computation graph of the first model at the host end.

[0013] In one possible implementation, the computation graph of the first model and the splitting rules of the optimizer are the same.

[0014] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0016] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, a cluster of computing devices may include one or more computing devices.

[0017] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the evolution of a model training technology concept provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a model training architecture provided in an embodiment of this application;

[0020] Figure 3 This is a comparative schematic diagram of a model training architecture provided in an embodiment of this application;

[0021] Figure 4 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram illustrating a comparison of loss overflow judgment time points provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0026] Figure 9This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0027] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0028] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0029] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0031] First, the relevant technical terms involved in the technical solution provided in this application will be introduced.

[0032] (1) Neural Network Model

[0033] A neural network model is a computational model inspired by the structure of the human brain. It processes information by mimicking the connections and interactions of neurons in the brain. In a neural network model, the network layer is the basic unit that constitutes the model; it is a hierarchical structure for processing data. Each network layer can consist of multiple neurons, which are responsible for receiving input, performing calculations, and passing the results to the next layer. Furthermore, each network layer can be responsible for performing a specific computational task. Network layers in a neural network model can include one or more of the following: input layer, hidden layer, output layer, fully connected layer, convolutional layer, pooling layer, self-attention layer, normalization layer, and multilayer perceptron (MLP). For example, a neural network model can be, but is not limited to, a deep neural network (DNN) model, a convolutional neural network (CNN) model, or a transformer model.

[0034] (2) Host / Device heterogeneous

[0035] Host / Device heterogeneity refers to an architecture in a computing system where computational tasks are distributed to different types of processors for collaborative work. In this architecture, "Host" typically refers to the main processor (e.g., a central processing unit, CPU), responsible for handling logic control, task scheduling, and memory management; while "Device" refers to accelerated processors (e.g., graphics processing units, GPUs, tensor processing units, TPUs), responsible for executing large-scale parallel computing tasks. This design fully leverages the strengths of different processors, achieving efficient task processing and performance optimization.

[0036] (3) Host / Device parameter exchange

[0037] Host / Device parameter exchange refers to the process of data and instruction transfer between the host and device in a heterogeneous computing system via an interface or bus. This process mainly includes: transferring data to be processed from the host to the device so that the device can execute the corresponding computation task; sending the computation results back to the host from the device; and also involving the transfer of control instructions to guide the device in performing specific operations. This efficient exchange of data and instructions is one of the key factors in ensuring the performance of a heterogeneous computing system.

[0038] (4) Mixing accuracy

[0039] Mixed precision is a strategy used during neural network model training to apply different numerical precisions to different operations within the model. It combines the use of 16-bit (half-precision) and 32-bit (single-precision) floating-point numbers to accelerate execution and reduce memory consumption. It can significantly improve training efficiency with almost no impact on the final accuracy of the model.

[0040] (5) Model parallelism

[0041] Model parallelism is a technique that divides a large neural network model into multiple sub-models and distributes these sub-models to different devices or computing units for parallel computation. It aims to improve the speed and efficiency of model training while reducing memory usage and resource overhead.

[0042] (6) Parallel Pipeline

[0043] Pipeline parallelism is a technique that divides the model training process into multiple stages and executes these stages in parallel in a pipeline manner to improve the efficiency of large-scale model training.

[0044] (7) Optimizer splits parallelism

[0045] Optimizer splitting parallelism is a technique for optimizing memory and performance in distributed training. It distributes the computation of the optimizer across different computing devices in the data parallel dimension, where each device updates its parameters and then shares the update results through communication operations. It is particularly suitable for large-scale neural network models to effectively reduce memory consumption and improve network performance.

[0046] The technical solution provided in this application will be described below.

[0047] In the training of neural network models, to improve training efficiency and fully utilize the advantages of different hardware platforms, a Host / Device heterogeneous computing strategy can be adopted. When employing a Host / Device heterogeneous computing strategy, such as... Figure 1As shown in (A), the model and optimizer can be configured on the device side. In this case, forward (FWD) propagation computation, backward (BWD) propagation computation, and optimizer computation can be performed on the device side. However, the device side uses an accelerator processor, which has limited memory resources, making it easy to encounter memory shortage problems during model training. To avoid memory shortage problems, such as... Figure 1 As shown in (B), the optimizer used during model training can be configured on the host side. Figure 1 In the scheme shown in (B), after the device side completes the backpropagation calculation and obtains the parameter gradients each time, the optimizer calculation can be performed on the host side to update the model parameters. However, since the host side needs to wait for the device side to complete the backpropagation calculation before it can perform the optimizer calculation, it is easy for the host side to wait for the device side for a long time, resulting in low model training efficiency.

[0048] In view of this, embodiments of this application provide a model training method that can improve model training efficiency when using a Host / Device heterogeneous computing strategy for model training.

[0049] Please see Figure 1 (C), Figure 1 (C) illustrates a technical concept for model training provided in an embodiment of this application. Figure 1As shown in (C), an optimizer is configured on the Host side, and the model is configured on the Device side. On the Host side, the optimizer is grouped into multiple sub-optimizers. Each sub-optimizer can handle a portion of the model's parameters, meaning it independently updates a subset of the parameters. On the Device side, the model's computation graph is segmented into multiple sub-computation graphs (i.e., subgraphs), with each subgraph associated with a sub-optimizer. Under this technical concept, after backpropagation computation in a subgraph (hereinafter referred to as "subgraph A"), the parameter gradients obtained through subgraph A can be passed to the sub-optimizer associated with subgraph A, and also to another subgraph (hereinafter referred to as "subgraph B") that has a data dependency on subgraph A, for backpropagation computation in subgraph B. In this way, subgraph B and the sub-optimizers associated with subgraph A can be computed in parallel. That is, while performing backpropagation computation in subgraph B, the computation of the sub-optimizers associated with subgraph A is also performed, or vice versa. Since each sub-optimizer is responsible for a portion of the model's parameters, and the execution of the sub-optimizers can be parallelized with the backpropagation computation of the model's subgraphs, different sub-optimizers do not need to wait for each other to execute. Therefore, the execution efficiency of the optimizers on the host side can be significantly improved, thereby improving the model training efficiency. (Continue reading...) Figure 1 In (C), the optimizer is divided into N sub-optimizers on the host side, and the computational graph of the model is divided into N subgraphs on the device side. The backpropagation computation performed in subgraph N is the BWD N shown in the diagram, and sub-optimizer N is associated with subgraph N. Figure 1 As shown in (C), the parameter gradients calculated by subgraph N can be transmitted to sub-optimizer N and subgraph N-1 (i.e., the subgraph corresponding to BWD N-1 in the figure), respectively. Subgraph N-1 and sub-optimizer N can be computed in parallel. In this embodiment, when there are multiple devices on the Device side, the computation graph of the model configured on each device can be segmented.

[0050] To facilitate understanding, the following example illustrates how a device on the Device side collaborates with the Host during model training. Figure 1 The technical concept shown in (C) will be introduced.

[0051] like Figure 2 As shown, on the Host, the optimizer is divided into N sub-optimizers. Simultaneously, on the Device, the computation graph of the configured model is divided into N sub-optimizers. A sub-graph on the Device is associated with a sub-optimizer on the Host. On the Device, FWD represents forward propagation computation, and BWD represents backward propagation computation. FWD 1 and BWD 1 are associated through sub-optimizers. Figure 1 To achieve this, FWD 2 and BWD 2 are implemented through sub- Figure 2 In this implementation, FWD N and BWDN are achieved through subgraph N. After calculating the loss (Loss) via forward propagation on the device, backward propagation can be performed. During backward propagation, the parameter gradients calculated in the i-th subgraph can be passed to the i-th sub-optimizer and the (i-1)-th subgraph. The (i-1)-th subgraph and the i-th sub-optimizer can be computed in parallel. During backward propagation, the input to the N-th subgraph is the Loss calculated via forward propagation.

[0052] In addition, such as Figure 3 As shown, when Device is an NPU and Host is a CPU, if using Figure 1 As shown in (B) of the Host / Device heterogeneous computing strategy, the backpropagation computation on the NPU and the optimizer execution on the CPU can be performed as follows: Figure 3 As shown in (A); using Figure 1 As shown in (C) of the Host / Device heterogeneous computing strategy, the backpropagation computation on the NPU and the optimizer execution on the CPU can be performed as follows: Figure 3 As shown in (B). Figure 3 In (A), since the optimizer on the CPU is not split, the CPU starts executing the optimizer calculation and updating the model parameters after collecting all the backpropagation calculation results from the NPU. Figure 3 In (B), each sub-optimizer on the CPU collects the backpropagation computation results obtained from the subgraph computation associated with that sub-optimizer on the NPU, and then optimizes and updates the model parameters. Different sub-optimizers do not need to wait for each other.

[0053] In the above process, by adopting a Host / Device heterogeneous computing strategy, the model can be trained efficiently on multiple GPUs and CPUs, while reducing memory consumption and improving training speed. By dividing the optimizer configured on the Host side into multiple groups (i.e., multiple sub-optimizers), each group is responsible for a portion of the model's parameters and can independently perform its parameter updates. This reduces communication overhead between different sub-optimizers and improves parallel efficiency. During the model's backpropagation, computational tasks are distributed to different processing units (such as GPUs or TPUs), with each unit responsible for calculating a portion of the gradients. Efficient data synchronization mechanisms, such as ensemble communication or pipeline synchronization, ensure that all processing units have complete gradient information before updating parameters. Furthermore, by optimizing the scheduling of computational tasks, Host-side computation can be hidden within Device-side computation time; that is, optimization computation can be parallelized with backpropagation computation, thereby improving overall training efficiency.

[0054] The following describes the specific implementation process of the above technical concept.

[0055] Figure 4 This diagram illustrates a model training method according to an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 4 As shown, the model training method may include the following steps:

[0056] S401. Configure the optimizer of the first model to be trained on the Host side, and split the optimizer of the first model to obtain N (N≥2) sub-optimizers.

[0057] In this embodiment, the optimizer for the first model to be trained can be configured on the host side. This optimizer can be pre-installed on the host side or uploaded by the user; the specific configuration depends on the actual situation and is not limited here. When the optimizer for the first model exists on the host side, it can be understood that the optimizer for the first model has been configured on the host side.

[0058] Furthermore, on the host side, the optimizer of the first model is split according to the grouping configuration rules to obtain N (N≥2) sub-optimizers. Each sub-optimizer independently performs a portion of the parameter updates for the first model. The grouping configuration rules may include: the size of the grouped parameters, the value of N, etc. The grouping configuration rules can be pre-defined or uploaded by the user. In some embodiments, the host side can primarily be used to host the model's optimizer. Additionally, the host side can also be used to perform model parameter updates, model splitting, etc.

[0059] S402. On the Host side, the computation graph of the first model to be trained is split into N sub-computation graphs, where each sub-computation graph is associated with a sub-optimizer.

[0060] In this embodiment, the computation graph of the first model can be divided into N sub-computation graphs on the host side according to the same grouping configuration rules as the optimizer. Each sub-computation graph is associated with a sub-optimizer, meaning that one sub-optimizer can independently be responsible for updating the parameters in one sub-computation graph.

[0061] S403. Configure N sub-computation graphs to the Device.

[0062] In this embodiment, after the computation graph of the first model is partitioned on the host side, the N sub-computation graphs can be distributed from the host side to the device side. This allows the N sub-computation graphs to be configured on the device side, enabling forward and backward propagation computations to be performed on the device side during the training of the first model. In some embodiments, the device side is primarily used to host the model and perform model computations. In some embodiments, this step can also be understood as configuring the first model on the device side.

[0063] S404. The first model is trained through collaboration between the Host and Device ends, wherein during the training process, at least one sub-computation graph is executed in parallel with at least one sub-optimizer in the backpropagation computation.

[0064] In this embodiment, after the Device receives the sub-computation graph from the Host, the Device can perform forward and backward propagation calculations for the first model. During backward propagation, after calculating the parameter gradient through any sub-computation graph, the Device can transmit the calculated parameter gradient to the Host and then to the next sub-computation graph for backward propagation. Upon receiving a parameter gradient from the Device, the Host can use the sub-optimizer associated with the sub-computation graph that calculated the gradient to process the gradient and obtain the model update parameters associated with that sub-optimizer. For example, the Device can first perform forward propagation calculations. Then, according to the data dependencies between the N sub-computation graphs, backward propagation calculations are performed sequentially, and the calculation results are transmitted to the Host. If there is a data dependency between the i-th sub-computation graph and the (i-1)-th sub-computation graph in N sub-computation graphs, then after the Device calculates the gradient of the i-th parameter through the i-th sub-computation graph, it can transmit the gradient of the i-th parameter to the Host. Simultaneously, the Device can perform backpropagation computation through the (i-1)-th sub-computation graph. While the Device is performing backpropagation computation through the (i-1)-th sub-computation graph, the Host can process the gradient of the i-th parameter through the i-th sub-optimizer associated with the i-th sub-computation graph to obtain the i-th updated parameter. For example, after the Host calculates an updated parameter through one of its sub-optimizers, it can transmit this updated parameter to the Device so that the Device can update the parameters of its model. Since a sub-optimizer is responsible for the parameters in one sub-computation graph of the model, after receiving the updated parameter calculated by a sub-optimizer, the Device can update the parameters in the sub-computation graph associated with that sub-optimizer in the model.

[0065] In this way, by configuring the model on the Device side and dividing the model's computation graph into multiple parts, and configuring the optimizer on the Host side and dividing the optimizer into multiple parts, with each sub-optimizer responsible for independently updating the parameters in a sub-computation graph, the optimizer computation and backpropagation computation can be parallelized, thereby improving the model training efficiency.

[0066] In some embodiments, Figure 4 S402 in the above can also be executed on the device side. In this case, S402 and S403 can be replaced by configuring the first model on the device side, where the computation graph of the first model is divided into N sub-computation graphs.

[0067] In some embodiments, to further improve model training speed and reduce memory usage, mixed-precision training techniques can be employed during model training. Mixed-precision training primarily uses FP16 data types instead of FP32 for data storage, reducing memory usage and increasing computational speed. However, due to the small representation range of FP16, underflow may occur when parameters (such as gradients) become very small during training, causing the model to fail to update weights correctly and affecting training performance. To reduce the probability of this happening, loss scaling techniques can be used. Loss scaling multiplies the loss by a large scaling factor when calculating the loss, thus amplifying the gradient during backpropagation. Due to the chain rule, the amplified loss leads to amplified gradients, thereby avoiding numerical underflow in FP16 representation. However, loss scaling itself can also encounter overflow issues. If the amplified gradient value exceeds the maximum value that FP16 can represent during training, overflow will occur. To avoid this, the value of loss scaling needs to be dynamically adjusted. If an overflow is detected, the value of Loss Scale will decrease. If no overflow occurs within a certain period of time, the value of Loss Scale will be increased to balance the amplified gradient and avoid the risk of overflow.

[0068] When determining Loss Scale overflow, such as Figure 5 As shown in (A), loss scale overflow detection can be performed after all parameter gradient calculations and before model parameter updates. That is, loss scale overflow detection is performed before the optimizer is executed on the host side. However, this loss scale overflow detection method prevents backpropagation calculations and the optimizer from being executed in parallel. Therefore, in this embodiment, as... Figure 5As shown in (B), the optimizer can be executed before the Loss Scale overflow check, allowing backpropagation computation to run in parallel with the optimizer. In this case, after the updated parameters are calculated by each sub-optimizer, the decision to assign them to the training model can be made after the Loss Scale overflow check. For example, the Loss Scale overflow check can be executed on either the host or the device; this is not limited here. When executed on the device, the device can transmit the check result to the host.

[0069] In some embodiments, the product implementation of the model training method provided in this embodiment can be included in deep learning platform software as deep learning training optimization software. This software can run on a general-purpose computer without relying on specific hardware devices. Such software can be deployed on various server hardware, including but not limited to high-performance servers in data centers, cloud computing platforms, or personal workstations, as long as these devices have the capability to run deep learning frameworks and process large amounts of data. During runtime, the program code of the model training method runs on both the host and device sides of the server.

[0070] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0071] Based on the methods in the above embodiments, this application also provides a model training device.

[0072] For example, Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of this application is shown. Figure 6 As shown, the model training device 600 includes a configuration module 601 and a training module 602. The configuration module 601 is used to configure the first model to be trained on the Device side, wherein the computation graph of the first model is divided into N sub-computation graphs, N≥2; and to configure the optimizer of the first model on the Host side, and to divide the optimizer into N sub-optimizers, wherein each sub-optimizer is responsible for independently updating the parameters in one sub-computation graph. The training module 602 is used to train the first model collaboratively on the Host and Device sides, wherein during the training process, at least one sub-computation graph is executed in parallel with at least one sub-optimizer in the backpropagation computation.

[0073] In some embodiments, the training module 602 is further configured to: employ mixed-precision training during the training of the first model, and, after the optimizer on the host has finished executing, perform a loss scaling overflow judgment.

[0074] In some embodiments, the N sub-computation graphs are obtained by dividing the computation graph of the first model at the host end.

[0075] In some embodiments, the computation graph of the first model and the partitioning rules of the optimizer are the same.

[0076] In some embodiments, Figure 6 Both the configuration module 601 and the training module 602 shown can be implemented in software or in hardware. For example, the implementation of the configuration module 601 will be described below. Similarly, the implementation of the training module 602 can be referenced from the implementation of the configuration module 601.

[0077] As an example of a software functional unit, configuration module 601 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, configuration module 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0078] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0079] As an example of a hardware functional unit, configuration module 601 may include at least one computing device, such as a server. Alternatively, configuration module 601 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0080] The multiple computing devices included in configuration module 601 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in configuration module 601 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in configuration module 601 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0081] It should be noted that, in other embodiments, the configuration module 601 can be used to execute any step in the model training method described in the above embodiments, and the training module 602 can be used to execute any step in the model training method described in the above embodiments. Furthermore, the steps implemented by the configuration module 601 and the training module 602 can be specified as needed, and different steps in the model training method described in the above embodiments can be implemented by the configuration module 601 and the training module 602 respectively. Figure 6 The model training device 600 shown has all the functions shown.

[0082] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0083] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0084] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0085] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0086] The memory 706 stores executable program code, and the processor 704 executes the executable program code to implement the aforementioned functions respectively. Figure 6 The configuration module 601 and training module 602 shown herein perform their functions to implement the model training method described in the above embodiments. That is, the memory 706 stores instructions for executing the model training method described in the above embodiments.

[0087] Alternatively, the memory 706 stores executable code, and the processor 704 executes the executable code to implement the aforementioned functions respectively. Figure 6 The model training apparatus 600 shown in the diagram performs the functions of the model training method described in the above embodiments. That is, the memory 706 stores instructions for executing the model training method described in the above embodiments.

[0088] The communication interface 703 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0089] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0090] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same instructions for executing the model training method described in the above embodiments.

[0091] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the model training method described in the above embodiments. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing the model training method described in the above embodiments.

[0092] It should be noted that the memory 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 6 The model training device 600 shown contains some of the functions. That is, the instructions stored in the memory 706 in different computing devices 700 can implement the functions of one or more modules in the configuration module 601 and the training module 602.

[0093] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, the two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for executing the functions of configuration module 601. Simultaneously, the memory 706 in computing device 700B stores instructions for executing the functions of training module 602.

[0094] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0095] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different in that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0096] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the aforementioned methods. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the aforementioned methods.

[0097] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0098] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0099] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0100] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0101] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0102] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: Configure the first model to be trained on the device, wherein the computation graph of the first model is divided into N sub-computation graphs, where N≥2; Configure the optimizer of the first model on the host side, and split the optimizer to obtain N sub-optimizers, wherein one of the sub-optimizers is responsible for independently executing the parameter update of one of the sub-computation graphs; The first model is trained collaboratively by the Host and the Device, wherein, during the training process, at least one of the sub-computation graphs is executed in parallel with at least one of the sub-optimizers in the backpropagation computation.

2. The method of claim 1, wherein, The method further includes: Mixed-precision training is used during the training of the first model, and loss scaling overflow is checked after the optimizer on the host has finished executing.

3. The method according to claim 1 or 2, characterized in that, The N sub-computation graphs are obtained by segmenting the computation graph of the first model at the Host end.

4. The method according to any of claims 1 to 3, characterized in that, The computational graph of the first model and the splitting rules of the optimizer are the same.

5. A model training apparatus characterized by comprising: include: The configuration module is used to configure the first model to be trained on the device, wherein the computation graph of the first model is divided into N sub-computation graphs, where N≥2; The configuration module is also used to configure the optimizer of the first model on the host side and split the optimizer to obtain N sub-optimizers, wherein one of the sub-optimizers is responsible for independently executing the parameter update of one of the sub-computation graphs; The training module is used to train the first model in collaboration between the Host and the Device, wherein, during the training process, at least one of the sub-computation graphs is executed in parallel with at least one of the sub-optimizers in the backpropagation computation.

6. The apparatus of claim 5, wherein, The training module is also used for: Mixed-precision training is used during the training of the first model, and loss scaling overflow is checked after the optimizer on the host has finished executing.

7. The apparatus of claim 5 or 6, wherein, The N sub-computation graphs are obtained by segmenting the computation graph of the first model at the Host end.

8. The apparatus according to any one of claims 5-7, characterized in that, The computational graph of the first model and the splitting rules of the optimizer are the same.

9. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-4, wherein the cluster of computing devices includes at least one computing device.

Images