Model training method and apparatus, and computing device

Abstract

The invention provides a model training method. The method comprises the steps of collecting information of a first model; and making a re-calculation strategy according to the communication time of the first model in the communication stage and the execution time required by the plurality of operators in the first model. The information of the first model comprises execution time respectively required by a plurality of operators in the first model, the recalculation strategy comprises at least one recalculation operator in the plurality of operators and opportunity for executing a recalculation process by the at least one recalculation operator, the recalculation operator is an operator used for executing the recalculation process in the plurality of operators, and the opportunity for executing the recalculation process by the at least one recalculation operator in the plurality of operators is the opportunity for executing the recalculation process by the at least one recalculation operator. The opportunity at which the at least one recalculation operator performs the recalculation process includes performing the recalculation process in parallel with the communication phase of the first model. The recalculation process is parallel to the communication process of the model, so that the model training time is shortened, and the throughput of model training is improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410598417.6, filed on May 14, 2024, with the Chinese National Intellectual Property Administration, entitled “Method, Apparatus and Computing Device for Model Training”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of AI, and more specifically, to a method, apparatus, and computing device for model training. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI models, also known as AI algorithms, are a collective term for mathematical algorithms built upon the principles of artificial intelligence and form the foundation for solving specific problems using AI. Before any AI model can be used to solve a specific technical problem, it needs to be trained. AI model training refers to using a specified initial model to calculate training data, and then adjusting the parameters of the initial model based on the calculation results, allowing the model to gradually learn certain patterns and acquire specific functions. After training, an AI model with stable functionality can be used for inference.

[0004] In recent years, with the rapid development of AI models, the parameter scale of these models has become increasingly large, further amplifying their demand for GPU memory. GPU memory capacity has gradually become a bottleneck restricting the training performance of large models. Recomputation is a commonly used GPU memory optimization technique, which refers to reducing the amount of GPU memory occupied by discarding some intermediate results generated during the computation process. One related recomputation scheme discards the activation data generated during the forward propagation (FP) computation of the model training and generates this activation data through an additional FP computation before the backpropagation (BP) computation. In this recomputation scheme, recalculating this activation data increases the model training time, thereby reducing the throughput of model training.

[0005] Therefore, how to shorten the model training time and increase the throughput of model training has become an urgent technical problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, and computing device for model training. The method aims to optimize and improve the data processing capabilities of the entire training process by parallelizing the recomputation process with the model's communication process, thereby shortening the model training time and increasing the throughput of model training.

[0007] Firstly, a method for model training is provided, comprising: collecting information about a first model; and formulating a recomputation strategy based on the communication time of the first model during a communication phase and the execution time required by multiple operators in the first model. The information about the first model includes the execution time required by multiple operators in the first model, and the recomputation strategy includes: at least one recomputation operator among the multiple operators, and the timing at which the at least one recomputation operator executes the recomputation process. The recomputation operator is the operator among the multiple operators used to execute the recomputation process, and the timing at which the at least one recomputation operator executes the recomputation process includes: executing the recomputation process in parallel with the communication phase of the first model.

[0008] In the above technical solution, the costly recomputation process is reasonably scheduled into a communication process through a recomputation strategy and executed in parallel with the communication process. This can make full use of the idle computing resources in the communication process for recomputation, thereby reducing the overhead of the recomputation process. When training the model on a device with limited video memory resources, the data processing capability of the entire training process is optimized and improved, thereby shortening the model training time and increasing the throughput of model training.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the information of the first model also includes the dependencies between the plurality of operators, including the first computation operator depending on the second computation operator, and the first computation operator performing the recomputation process based on the output of the second computation operator when the output of the second computation operator is stored in the video memory.

[0010] In the above technical solution, based on the dependency relationship between operators, the dependent recomputation operator (e.g., the first recomputation operator) can complete the above recomputation process only if the output of the dependent recomputation operator (e.g., the second recomputation operator) is stored in the video memory.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the at least one recomputation operator includes at least one target recomputation operator that executes the recomputation process in parallel with the first communication phase, wherein the execution time required for the at least one target recomputation operator to execute the recomputation process is less than or equal to the communication time of the first communication phase.

[0012] In the above technical solution, the execution time required for at least one target recalculation operator that performs the recalculation process in parallel with a certain communication stage should be less than or equal to the communication time of the first communication stage. This can further shorten the model training time and improve the throughput of model training.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the information of the first model also includes the data size output by each of the plurality of operators, wherein the sum of the video memory capacity occupied by the data output by the at least one recalculation operator in performing the recalculation process and the video memory capacity occupied by the data in resident video memory is less than or equal to the total capacity of the video memory.

[0014] In the above technical solution, since the peak value of the video memory during model training cannot exceed the total capacity of the video memory, the sum of the video memory capacity occupied by the data output by the recalculation operator during the recalculation process and the video memory capacity occupied by the data in the resident video memory should be less than or equal to the total capacity of the video memory.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the information of the first model also includes the type of each operator, wherein the type of the operator includes a communication operator and a computation operator, and the at least one recomputation operator does not include the communication operator.

[0016] In the above technical solution, since the communication operator occupies communication resources, the communication operator cannot be executed in parallel during the communication process. That is, the communication operator is not included in at least one of the recalculation operators that execute the recalculation process in parallel with the communication process.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, formulating a recomputation strategy based on the communication time of the first model during the communication phase and the information of the first model includes: constructing an objective function, the objective function being the total execution time required for the at least one recomputation operator to perform the recomputation process, the independent variable of the objective function including the information of the first model; determining the dependent variable of the objective function based on the objective task, wherein the dependent variable of the objective function is used to indicate the recomputation strategy, the objective task being to minimize the total execution time.

[0018] In the above technical solution, the above recomputation strategy can be formulated by finding the optimal solution through minimizing the objective function.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each of which has at least one server.

[0020] Secondly, a model training apparatus is provided, comprising: an information acquisition module, a strategy formulation module, and a strategy execution module. The information acquisition module is used to acquire information about a first model, including the execution time required for each of multiple operators in the first model. The strategy formulation module is used to formulate a recomputation strategy based on the communication time of the first model during a communication phase and the execution time required for each of the multiple operators in the first model. The recomputation strategy includes: at least one recomputation operator among the multiple operators and the timing at which the at least one recomputation operator executes the recomputation process. The recomputation operator is the operator among the multiple operators used to execute the recomputation process, and the timing at which the at least one recomputation operator executes the recomputation process includes: executing the recomputation process in parallel with the communication phase of the first model. The strategy execution module is used to train the first model according to the recomputation strategy.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, the information of the first model also includes the dependencies between the multiple operators, including the first computation operator depending on the second computation operator, and the first computation operator performing the recomputation process based on the output of the second computation operator when the output of the second computation operator is stored in the video memory.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the at least one recomputation operator includes at least one target recomputation operator that executes the recomputation process in parallel with the first communication phase, wherein the execution time required for the at least one target recomputation operator to execute the recomputation process is less than or equal to the communication time of the first communication phase.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the information of the first model also includes the data size output by each of the plurality of operators, wherein the sum of the video memory capacity occupied by the data output by the at least one recalculation operator in performing the recalculation process and the video memory capacity occupied by the data in resident video memory is less than or equal to the total capacity of the video memory.

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the information of the first model also includes the type of each operator, wherein the type of the operator includes a communication operator, a computation operator, and the at least one recomputation operator does not include the communication operator.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the strategy formulation module is specifically used to: construct an objective function, which is the total execution time required for the at least one recomputation operator to perform the recomputation process, the independent variables of the objective function including information of the first model; and determine the dependent variable of the objective function according to the objective task, wherein the dependent variable of the objective function is used to indicate the recomputation strategy, the objective task being to minimize the total execution time.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, the device is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each of which has at least one server.

[0027] It should be understood that the beneficial effects of the second aspect and its various implementations can be found in the first aspect and its various implementations, and will not be elaborated here.

[0028] Thirdly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the program to execute the method of the first aspect or any possible implementation thereof.

[0029] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0030] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method of the first aspect or any possible implementation thereof.

[0031] Fifthly, a chip is provided that acquires and executes instructions to implement the methods described in the first aspect and any implementation thereof.

[0032] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect and any implementation thereof.

[0033] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect and any implementation thereof.

[0034] In a sixth aspect, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first aspect and any implementation thereof.

[0035] In a seventh aspect, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and any implementation thereof.

[0036] Eighthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first aspect and any implementation thereof.

[0037] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0038] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium.

[0039] A ninth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method as described in the first aspect and any implementation thereof.

[0040] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0041] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0042] Figure 1 It is a schematic diagram illustrating the training and inference processes of a model.

[0043] Figure 2 This is a schematic block diagram of a cloud scenario applicable to embodiments of this application.

[0044] Figure 3 This is a schematic block diagram of a system architecture applicable to embodiments of this application.

[0045] Figure 4 This is a diagram illustrating one way multiple GPUs can be connected within a GPU cluster.

[0046] Figure 5 This is a diagram illustrating another way to connect multiple GPUs within a GPU cluster.

[0047] Figure 6 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application.

[0048] Figure 7 This is a schematic block diagram of a model training apparatus 700 provided in an embodiment of this application.

[0049] Figure 8 This is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application.

[0050] Figure 9 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.

[0051] Figure 10 This is a schematic diagram showing the connection between computing devices 1500A and 1500B via a network, as provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0053] This application will present various aspects, embodiments, or features relating to systems comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0054] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0055] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0056] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0059] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0060] The basic principle of AI is to combine massive amounts of data with powerful computing capabilities and intelligent algorithms to build an AI model that solves specific problems. This AI model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking that is close to that of humans.

[0061] AI models, also known as AI algorithms, are a collective term for mathematical algorithms built upon the principles of artificial intelligence. They form the foundation for using AI to solve specific problems. Depending on the specific methods and / or technologies used to implement artificial intelligence, AI models can also be called machine learning models, deep learning models, or reinforcement learning models.

[0062] Before any AI model can be used to solve a specific technical problem, it needs to be trained. AI model training refers to using a specified initial model to compute on training data, and then adjusting the parameters of the initial model based on the computation results, so that the model gradually learns certain patterns and acquires specific functions. Once trained and possessing stable functionality, the AI ​​model can be used for inference. AI model inference is the process of using the trained AI model to compute on input data and obtain predicted inference results.

[0063] The most common approach is supervised training of AI models. For example, most deep learning models are trained using supervised training methods. The following section will combine... Figure 1 This section introduces the most widely used supervised training methods for deep learning models.

[0064] like Figure 1 As shown, during the training phase, a training set for the deep learning model needs to be constructed based on the objective. The training set includes multiple training data points, each labeled. The label of a training data point represents the correct answer to a specific question, and the label can indicate the objective of training the deep learning model using the training data. For example, to train a deep learning model that can identify different animals, the training set can include images of multiple different animals (i.e., training data). Each image can have a label identifying the type of animal it contains, such as cat or dog. In this example, the type of animal corresponding to each image is the label of that training data.

[0065] When training a deep learning model, training data can be input into the model in batches after parameter initialization. The deep learning model performs calculations (i.e., inference) on the training data to obtain prediction results based on the training data. The prediction results obtained through inference, along with the corresponding labels of the training data, are used as data to calculate the loss based on the loss function. The loss function is used during the model training phase to calculate the difference (i.e., the loss value) between the model's prediction results on the training data and the labels of that training data. The loss function can be implemented using different mathematical functions; commonly used expressions for loss functions include: mean squared error loss function, logarithmic loss function, least squares method, etc.

[0066] The loss value calculated based on the loss function can be used to update the parameters of a deep learning model. Gradient descent is a common method for updating these parameters. Model training is an iterative process where each iteration infers from different training data and calculates the loss value. The goal of these multiple iterations is to continuously update the parameters of the deep learning model and find the parameter configuration that minimizes or stabilizes the loss value of the loss function.

[0067] It should be understood that a loss function is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, loss functions are often used as learning criteria and associated with optimization problems; that is, the model is solved and evaluated by minimizing the loss function. For example, in machine learning, loss functions are used for parameter estimation of the model, and the loss value obtained based on the loss function can be used to describe the degree of difference between the model's predictions and the actual values. Common loss functions include the mean squared error loss function, the support vector machine (SVM) hinge loss function, and the cross-entropy loss function.

[0068] During the training phase, to improve training efficiency and post-training model performance, it's necessary to set appropriate hyperparameters. Hyperparameters in deep learning models refer to parameters that cannot be obtained through learning from training data or that cannot be changed by training data; they are a concept relative to the parameters in the model. Hyperparameters of deep learning models are typically set manually based on experience or experiments. These hyperparameters include: learning rate, batch size, and network structure hyperparameters (e.g., number of layers (also called depth), interaction methods between layers, number and size of convolutional kernels, activation functions, etc.). Among these, the learning rate, as a hyperparameter, controls the magnitude of parameter weight updates during training, significantly impacting training speed and accuracy.

[0069] like Figure 1 As shown, a trained deep learning model can be used to infer from input data. In the inference phase, data from real-world application scenarios is typically used as input. The trained deep learning model then infers the results. The inference phase is the practical application of the trained deep learning model, enabling the rapid use of AI capabilities to solve specific technical problems. Currently, AI has numerous application scenarios, and the inference capabilities of deep learning models can be applied to various scenarios, such as personnel recognition in access control and security systems, video surveillance for violence detection, and express delivery tracking number detection and recognition.

[0070] Deep learning models can include forward propagation (FP) computation and back propagation (BP) computation. The FP and BP computation processes are described in detail below.

[0071] In the process of FP calculation, training data is used as the input to the input layer of the deep learning model. This input passes through multiple neurons in the hidden layers and outputs a prediction result from the output layer. Specifically, each neuron in the hidden layer corresponds to a parameter matrix. The product of the input to the input layer and the parameter matrix of the first-layer neuron serves as the input to the first-layer neuron in the hidden layer. This input is then processed by the activation function (e.g., the sigmoid function) of the first-layer neuron, resulting in its output value. The product of the output value of the first-layer neuron and the parameter matrix of the second-layer neuron serves as the input to the second-layer neuron in the hidden layer. Similarly, this process continues until a prediction result is finally output from the output layer.

[0072] During backpropagation (BP) calculation, the error between the predicted value generated during front-factor (FP) calculation and the prior knowledge can be calculated. Then, along the output layer, hidden layer, and input layer, the weights in the parameter matrix of each neuron in the deep learning model are corrected based on this error. Specifically, the weights are corrected by adjusting the gradients of the weights in the parameter matrix for each neuron, and the predicted output value is calculated based on the corrected parameter matrix. Through multiple iterations, the weights in the parameter matrix are continuously adjusted, thereby ensuring that the output value predicted by the deep learning model is as close as possible to the prior knowledge of the training data.

[0073] It should be understood that the above only introduces the training of the most typical deep learning model as an example. The training of other types of models has slight differences, but the principle is similar. Most of them involve inference on the training data and adjusting the parameters in the model based on the inference results, with the goal of obtaining a parameter combination that makes the model performance stable.

[0074] In recent years, with the rapid development of AI models, the number of model parameters has been increasing dramatically. From 2020 to 2024, the number of parameters in known models increased more than tenfold, further amplifying the demand for GPU memory. GPU memory has gradually become a bottleneck restricting the training performance of large models. How to effectively utilize limited GPU memory during model training while achieving the highest throughput is an important research goal for current large model training systems.

[0075] It should be understood that the above-mentioned large model refers to a model with a large number of parameters.

[0076] Recomputation is a commonly used memory optimization technique that reduces memory usage by discarding intermediate results generated during computation. For example, in AI model training, memory consumption primarily comes from model parameters, gradients, and intermediate data (e.g., activation (AC) data), with activation data accounting for a significant proportion. Therefore, activation data generated during the initial computation (FP) of model training can be discarded and generated through an additional FP computation before the backpropagation (BP) computation.

[0077] In the recalculation process described above, although the amount of GPU memory used can be reduced by discarding activation data, the corresponding cost is that recalculating the activation data will increase the model training time, thereby reducing the throughput of model training.

[0078] In view of this, embodiments of this application provide a model training method that, when training a model on a device with limited video memory resources, focuses on optimizing and improving the data processing capabilities of the entire training process, thereby shortening the model training time and increasing the throughput of model training.

[0079] In one possible implementation, the method provided in this application embodiment can be applied to cloud service scenarios. For ease of description, the following will first combine... Figure 2 It provides a detailed description of cloud service scenarios.

[0080] Figure 2 This is a schematic block diagram illustrating a cloud scenario applicable to embodiments of this application. For example... Figure 2 As shown, the cloud scenario may include: cloud management platform 110, Internet 120, and client 130.

[0081] like Figure 2 As shown, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources to provide corresponding cloud services to tenants.

[0082] The cloud management platform 110 can be located in a cloud data center and provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110 and log in. After successful authentication of the cloud account and password by the cloud management platform 110, the tenant can further select and purchase virtual machines of specific specifications (processor, memory, disk) on the cloud management platform 110. After successful purchase, the cloud management platform 110 provides the remote login account and password for the purchased virtual machine, and client 130 can remotely log in to the virtual machine to install and run the tenant's applications. Therefore, tenants can create, manage, log in to, and operate virtual machines in the cloud data center through the cloud management platform 110. Virtual machines can also be referred to as Elastic Compute Service (ECS) or Elastic Instances (different cloud service providers may use different names).

[0083] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0084] The cloud management platform 110 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants can log in to the cloud management platform 110 via client 130 and the internet 120 to manage their rented cloud services.

[0085] For ease of description, the system architecture applicable to the embodiments of this application will be described first below.

[0086] Figure 3 This is a schematic block diagram of a system architecture applicable to embodiments of this application. For example... Figure 3 As shown, the system includes a central processing unit (CPU) and a graphics processing unit (GPU) cluster, which includes multiple GPUs.

[0087] The multiple GPUs within the aforementioned CPU and GPU cluster can be connected via a bus, which can be either a peripheral component interconnect express (PCI) bus or an extended peripheral component interconnect express (PCI Express, PCIE) bus. This application does not specifically limit the type of bus.

[0088] There are various ways to connect the multiple GPUs in the aforementioned GPU cluster, and this application embodiment does not specifically limit this. Two possible implementation methods are described below.

[0089] Implementation method 1, such as Figure 4 As shown, multiple GPUs within a GPU cluster are interconnected via a bus (e.g., a PCIe bus).

[0090] Implementation method 2, such as Figure 5 As shown, multiple GPUs within the GPU cluster are interconnected via NVIDIA Link (NVLink).

[0091] It should be understood that NVLink is a high-speed, low-latency GPU interconnect technology developed by NVIDIA to connect multiple GPU devices, improving parallel computing capabilities and data transfer efficiency. NVLink supports various data transfer rates and configurations and can be seamlessly integrated with NVIDIA's GPU architecture and other devices such as CPUs and memory.

[0092] The following is combined with Figure 6 This application provides a detailed description of a model training method based on embodiments. It should be understood that... Figure 6 The examples are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to... Figure 6 The specific numerical values ​​or specific scenarios illustrated. Those skilled in the art will understand based on... Figure 6 The examples given below can obviously be modified or varied in various ways, and such modifications and variations also fall within the scope of the embodiments of this application.

[0093] Figure 6 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. Figure 6 As shown, the method may include steps 610-630, which will be described in detail below.

[0094] Step 610: Collect information from the first model.

[0095] In this embodiment of the application, the configuration parameters of the first model can be obtained from the user, and the first model can be pre-trained according to the configuration parameters, and information of the first model can be collected during the pre-training process.

[0096] In one possible implementation, information from the first model can be collected through an information acquisition module, which can be deployed on a GPU.

[0097] As an example, before training the first model, the information acquisition module obtains the configuration parameters of the first model set by the user, and deploys the user-defined first model on the GPU device according to these configuration parameters. The GPU device's resources are then used to perform several rounds of pre-training iterations on the first model, and information about the first model is collected.

[0098] The configuration parameters of the first model mentioned above may include, but are not limited to, at least one of the following:

[0099] 1. Structural parameters of the first model, including but not limited to at least one of the following: number of layers in the first model, size of hidden layers, and number of attention heads.

[0100] 2. The strategy information for parallel training of the first model, including but not limited to at least one of the following: the settings information for the data parallelism (DP) group, the tensor parallelism (TP) group, and the pipeline parallelism (PP) group.

[0101] 3. Information on the resources for running the first model, including but not limited to at least one of the following: the number of GPUs running the first model, and the topological connection method between the GPUs.

[0102] 4. Hyperparameters of the first model, including but not limited to at least one of the following: learning rate, batch size, network structure hyperparameters (e.g., number of network layers (also known as depth), interaction between network layers, number and size of convolutional kernels, activation function, etc.).

[0103] As an example, the information collected from the first model may include the execution time of the model operators.

[0104] Optionally, the information collected for the first model may also include: the number of model operators, the number of model operators, the type of model operators, the size of the data output by the model operators, and the dependencies between model operators.

[0105] The aforementioned model operators refer to the operations in the model. A model consists of a series of operators, each with its own input and output tensors. The connections between operators form a directed graph, which reflects the computational order of different operations in the model, thus defining the computational flow of the model.

[0106] The types of operators mentioned above can include computation operators and communication operators. Common computation operators include, but are not limited to, convolution, fully connected layers, pooling, activation functions (such as ReLU and Sigmoid), batch normalization (such as BatchNormalization), and Dropout.

[0107] It should be understood that, for ease of description, the model operator can also be simply referred to as the operator below.

[0108] The dependency relationship between the operators in the above model refers to the dependency relationship between the input data and output data of the operators. For example, the input data of one operator depends on the output data of another operator.

[0109] For example, Table 1 shows an example of the information collected by the information collection module for the first model.

[0110] Table 1 provides an example of information from the first model.

[0111] Operator numbering Operator type Operator execution time Size of operator output data rely OP1 Comp 2ms 300MB / OP2 Comm 5ms 300MB OP1 OP3 Comp 1ms 300MB OP2

[0112] Step 620: Determine the recalculation strategy based on the information from the first model.

[0113] In this embodiment, a recomputation strategy can be formulated based on the information collected from the first model. One possible implementation involves using a recomputation strategy formulation module (also called a strategy formulation module), which can be deployed on a CPU.

[0114] The aforementioned recomputation strategies include, but are not limited to: recomputation operators among multiple operators, and the timing of recomputation initiation by recomputation operators. Here, a recomputation operator refers to the operator among the computational operators that requires recomputation. The timing of recomputation initiation by recomputation operators refers to the stage in training at which the recomputation operator recomputes its output data.

[0115] The timing for the aforementioned recomputation operator to initiate recomputation may include, but is not limited to: initiating recomputation in parallel during the communication process of a BP computation, initiating recomputation in parallel during the communication process of a FP computation, and initiating recomputation on the critical path.

[0116] It should be understood that initiating a recomputation on the critical path means that if a certain BP computation process requires certain discarded data, then that data is generated through an additional FP computation before that BP computation process.

[0117] As an example, parallel training of large models based on TP and PP has the following characteristics:

[0118] (1) Characteristic 1:

[0119] Training based on TP involves numerous communication operations, during which GPU computing resources are almost idle. Due to the use of SP or TP techniques, each round of forward or backward propagation iterations of the model involves two communications. When using TP alone, two All-Reduce operations are required, while when using TP and SP in combination, one All-Gather operation and one Reduce-Scatter operation are required.

[0120] (2) Feature 2:

[0121] The recomputation techniques in current mainstream frameworks are all placed on the critical path, that is, an extra forward propagation is performed to calculate the data before the backpropagation operation needs a certain data, which has a large overhead.

[0122] (3) Characteristic 3:

[0123] The memory requirements of each stage in the pipeline during PP training are different. Pipeline parallelism breaks the model down into N stages, with each stage deploying a portion of the model. For each micro-batch, the temporary data generated during forward propagation cannot be discarded before backpropagation ends. Therefore, earlier stages (with smaller numbers) need to store multiple intermediate results, resulting in higher storage requirements for earlier stages.

[0124] Because of the aforementioned characteristics of large-scale parallel training based on TP and PP, there is an opportunity to rationally schedule costly recomputation processes initiated on the critical path into a communication process. For example, by executing communication and recomputation processes in parallel during training, idle computing resources during communication can be fully utilized for recomputation, thereby reducing the overhead of recomputation. This optimizes and enhances the data processing capabilities of the entire training process when training models on devices with limited GPU memory, thus shortening training time and increasing training throughput.

[0125] In one implementation, this application designs a linear integer programming optimizer that can search for the optimal recomputation strategy based on at least one of the following parameters to minimize the total execution time of multiple operators: the number of operators, the type of operators, the execution time of operators, the size of the data output by operators, the dependencies between operators, the communication time required for each communication stage, and the capacity of GPU memory.

[0126] For example, one possible objective function is shown in formula (1):

[0127]

[0128] The objective function, constants, and variables in formula (1) will be explained in detail below.

[0129] f(S i ,R t,i ) represents the objective function, which represents the total execution time of n operators. The optimization objective of this application embodiment is to minimize the total execution time of n operators.

[0130] C i This represents the execution time of operator i.

[0131] S i This indicates whether the data output by operator i needs to reside in video memory, that is, whether operator i needs to undergo a recalculation process. i The value of is shown in formula (2), and its value includes 0 or 1. Wherein, S i A value of 0 indicates that the data output by operator i needs to remain resident in video memory and does not require recalculation; S i A value of 1 indicates that the data output by operator i will not remain resident in the video memory and will require recalculation.

[0132] S i ∈{0,1} (2)

[0133] R t,i R represents the process of whether operator i needs to be recalculated in stage t. t,i The values ​​of are shown in formula (3). The values ​​include 0 or 1. Wherein, R t,i A value of 0 indicates that operator i does not need to be recalculated in stage t; R t,i A value of 1 indicates that operator i needs to be recalculated in stage t.

[0134] Taking five stages as an example, the value of t is shown in formula (4), which includes any value from 0 to 4. Among them, a value of 0 indicates the first communication stage of forward propagation; a value of 1 indicates the second communication stage of forward propagation; a value of 2 indicates the first communication stage of back propagation; a value of 3 indicates the second communication stage of back propagation; and a value of 4 indicates the stage where an additional forward propagation calculation is performed before certain data is needed in a certain back propagation process (i.e., the critical path stage).

[0135] R t,i ∈{0,1} (3)

[0136] t∈{0, 1, 2, 3, 4} (4)

[0137] G t Let G be a constant. t The values ​​of t are shown in formula (5). When t takes any value from 0 to 3, G t The value of t is 0; when t takes any value from 0 to 3, G t The value of is 1.

[0138]

[0139] In this embodiment of the application, to avoid multiple recalculations in multiple stages of operator i, R can be adjusted using formula (6). t,i The values ​​of are constrained so that the R of operator i in the t stages are... t,i The sum is 1. That is, only in one of the t stages R... t,i The value of R is 1, and the remaining R values ​​in the t stages are... t,i The value of is 0.

[0140]

[0141] It should be understood that the embodiment of this application does not specifically limit the value of t. Formula (6) is used as an example of 5 stages for illustration.

[0142] In this embodiment of the application, if the dependency relationship between operators indicates that operator i depends on operator j, that is, if the input data of operator i depends on the output data of operator j, then whether operator i can be recalculated in stage t requires that the output data of operator j has been stored in the video memory in stage t. As an example, operator j needs to satisfy the following formula (7).

[0143]

[0144] (j,t)∈E represents a combination of dependencies between operators, where operator i depends on operator j;

[0145] R t',j This indicates whether operator j needs to undergo recalculation in stage t or a stage before stage t. A value of 0 indicates that operator j does not need to undergo recalculation in stage t or a stage before stage t; a value of 0 indicates that operator j needs to undergo recalculation in stage t or a stage before stage t.

[0146] S j Indicates whether the data output by operator j needs to be resident in video memory. A value of 0 means the data output by operator j does not need to be resident in video memory; a value of 1 means the data output by operator j needs to be resident in video memory.

[0147] It should be understood that in formula (7), if R t',j and S j If the value of is greater than or equal to 1, it can be understood that operator j has been over-calculated in stage t or in a stage before stage t, and / or the output data of operator j is resident in video memory.

[0148] In this embodiment of the application, the sum of the times of the recomputation operators that are executed in parallel with the communication process in stage t needs to be less than the communication time of stage t. The following uses four communication processes as an example to express the time of the recomputation operators that are executed in parallel with the communication process in stage t through formulas (8)-(11).

[0149]

[0150] In the above formula (8), R 0,i This indicates whether operator i needs to be recomputed during the first communication phase of the forward propagation; T ForComm1 This indicates the communication time of the first communication phase in the forward propagation.

[0151] In the above formula (9), R 1,i This indicates whether operator i needs to be recomputed during the second communication phase of the forward propagation; T ForComm2 This indicates the communication time of the second communication phase in the forward propagation.

[0152] In the above formula (10), R 2,i This indicates whether operator i needs to be recomputed during the first communication phase of backpropagation; T BankComm1 This indicates the communication time of the first communication phase in the forward propagation.

[0153] In the above formula (11), R 3,i This indicates whether operator i needs to be recomputed during the second communication phase of backpropagation; T BankComm2 This indicates the communication time of the first communication phase in the forward propagation.

[0154] In this embodiment, since the communication operator uses communication resources, it cannot be executed in parallel with the communication process. The following uses formula (12) to define the R of the communication operator. t,i The values ​​of are constrained.

[0155] R t,i =0, t∈{0,1,2,3},i∈COMM (12)

[0156] In formula (12) above, COMM represents the set of communication operators. If operator i belongs to the communication operators, the R of operator i... t,i The value of is 0.

[0157] In this embodiment of the application, in order to implement the recalculation process, the data output by the nth operator needs to be stored in the video memory as the starting data for recalculation. As an example, the stored data is constrained below in conjunction with formula (13).

[0158] S n ==1 (13)

[0159] In the above formula (13), S n This represents the data output by the nth operator. A value of 1 indicates that the data output by the nth operator needs to be stored in the video memory.

[0160] In this embodiment, since the capacity of the video memory is limited, the peak video memory usage during model training cannot exceed the device's video memory limit. For example, the video memory capacity occupied by resident data (including static data), the video memory capacity occupied by recomputed data, and the reserved video memory capacity (margin) should be less than or equal to the total video memory capacity. As an example, the following uses formula (14) to constrain the video memory capacity.

[0161]

[0162] M base This refers to static data that resides in GPU memory. For example, this static data may include, but is not limited to, model parameters, optimizer parameters, gradients, and other data.

[0163] NUM translayer This indicates the number of Transformer Layers in the current stage.

[0164] M i This indicates the storage space requirement corresponding to the output of operator i.

[0165] NUM stage This indicates the number of stages in the pipeline.

[0166] Mdelta This indicates the amount of GPU memory reserved to prevent out-of-memory (OOM) errors.

[0167] M constraint This indicates the total capacity of the GPU's video memory.

[0168] This indicates the amount of video memory occupied by data that resides in video memory.

[0169] This indicates the amount of video memory used by the recalculated data.

[0170] Step 630: Train the first model according to the established recomputation strategy.

[0171] In this embodiment, after the recomputation strategy formulation module executes the recomputation strategy, it can send the recomputation strategy to the strategy execution module. The strategy execution module can configure the recomputation strategy into the training strategy, and the first model deployed on the GPU executes the training strategy, thereby executing the recomputation strategy.

[0172] In one possible implementation, the policy execution module can be deployed on a GPU.

[0173] In the above technical solution, by rationally scheduling the costly recomputation process into a communication process through a recomputation strategy, and executing it in parallel with the communication process, the idle computing resources in the communication process can be fully utilized for recomputation, thereby reducing the overhead of the recomputation process. This optimizes and improves the data processing capability of the entire training process when training models on devices with limited video memory resources, thereby shortening the model training time and increasing the throughput of model training.

[0174] The above text combined Figures 1 to 6 The present application describes in detail the model training method provided in the embodiments. The following will combine... Figures 7-10 The embodiments of the apparatus of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0175] Figure 7 This is a schematic block diagram of a model training apparatus 700 provided in an embodiment of this application. The apparatus 700 can be implemented by software, hardware, or a combination of both. The apparatus 700 provided in this embodiment can implement the embodiments of this application. Figure 6The method flow shown includes the following: the device 700 comprises an information acquisition module 710, a strategy formulation module 720, and a strategy execution module 730. The information acquisition module 710 is used to acquire information about a first model, including the execution time required for each of the multiple operators in the first model. The strategy formulation module 720 is used to formulate a recomputation strategy based on the communication time of the first model during the communication phase and the execution time required for each of the multiple operators in the first model. The recomputation strategy includes at least one recomputation operator among the multiple operators and the timing at which the at least one recomputation operator executes the recomputation process. The recomputation operator is the operator among the multiple operators used to execute the recomputation process, and the timing at which the at least one recomputation operator executes the recomputation process includes executing the recomputation process in parallel with the communication phase of the first model. The strategy execution module 730 is used to train the first model according to the recomputation strategy.

[0176] Optionally, the information of the first model also includes the dependencies between the multiple operators, including the first computation operator depending on the second computation operator, and the first computation operator performing the recomputation process based on the output of the second computation operator when the output of the second computation operator is stored in the video memory.

[0177] Optionally, the at least one recomputation operator includes at least one target recomputation operator that executes the recomputation process in parallel with the first communication phase, wherein the execution time required for the at least one target recomputation operator to execute the recomputation process is less than or equal to the communication time of the first communication phase.

[0178] Optionally, the information of the first model also includes the data size output by each of the plurality of operators, wherein the sum of the video memory capacity occupied by the data output by the at least one recalculation operator performing the recalculation process and the video memory capacity occupied by the data resident in the video memory is less than or equal to the total capacity of the video memory.

[0179] Optionally, the information of the first model may also include the type of each operator, wherein the type of the operator includes a communication operator and a computation operator, and the at least one recomputation operator does not include the communication operator.

[0180] Optionally, the strategy formulation module 720 is specifically used to: construct an objective function, the objective function being the total execution time required for the at least one recomputation operator to perform the recomputation process, the independent variables of the objective function including information of the first model; and determine the dependent variable of the objective function according to the objective task, wherein the dependent variable of the objective function is used to indicate the recomputation strategy, the objective task being to minimize the total execution time.

[0181] Optionally, the device 700 is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, each of which has at least one server.

[0182] The device 700 here can be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0183] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of the information acquisition module 710 will be described below. Similarly, the implementation of other modules, such as the strategy formulation module 720 and the strategy execution module 730, can refer to the implementation of the information acquisition module 710.

[0184] As an example of a software functional unit, the information acquisition module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the information acquisition module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0185] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0186] As an example of a hardware functional unit, the information acquisition module 710 may include at least one computing device, such as a server. Alternatively, the information acquisition module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0187] The information acquisition module 710 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the information acquisition module 710 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the information acquisition module 710 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0188] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] It should be noted that the above embodiments of the device, when executing the above methods, are only illustrative examples of the division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the information acquisition module 710 can be used to execute any step in the above methods, the strategy formulation module 720 can be used to execute any step in the above methods, and the strategy execution module 730 can be used to execute any step in the above methods. The steps implemented by the information acquisition module 710, the strategy formulation module 720, and the strategy execution module 730 can be specified as needed. By implementing different steps in the above methods through the information acquisition module 710, the strategy formulation module 720, and the strategy execution module 730, all the functions of the above device can be realized.

[0190] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0191] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0192] The following is combined with Figure 8 This application provides a detailed description of a computing device provided in an embodiment.

[0193] Figure 8 This is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. The computing device 1500 can be a server, a computer, or other device with computing capabilities. Figure 8 The computing device 1500 shown includes at least one processor 1510 and a memory 1520.

[0194] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0195] The processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the method provided in this application. Alternatively, the processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0196] Optionally, the computing device 1500 also includes a communication interface 1530. The communication interface 1530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1500 and other devices or communication networks.

[0197] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, memory 1520, and communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540; for example, the processor 1510 can perform data read / write or code execution in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0198] In one possible implementation, the processor 1510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1520 or cache 1516.

[0199] Optionally, processor 1510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1510 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, etc. For example, processor 1510 may be a central processing unit (CPU).

[0200] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0201] Optionally, the processing unit 1512, also known as the core, is the most important component of the processor. The processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0202] In one implementation example, the memory control unit 1514 controls the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0203] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbitrator is configured in the system bus. Figure 8 (Not shown in the image), the arbitrator is responsible for handling and coordinating competing accesses of multiple processing units 1512.

[0204] In one implementation example, the processing unit 1512 and the memory control unit 1514 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1512 and the memory control unit 1514.

[0205] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When the processing unit 1512 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, its purpose is to help the processing unit 1512 run faster.

[0206] The memory 1520 provides runtime space for processes in the computing device 1500. For example, the memory 1520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0207] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1510 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1512 will send the result out.

[0208] By way of example and not limitation, memory 1520 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0209] The above-described structure of the computing device 1500 is merely illustrative and is not intended to limit the application. The computing device 1500 in this application includes various hardware components found in existing computer systems. For example, the computing device 1500 may also include other memories besides the memory 1520, such as disk storage. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1500 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device 1500 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 8 All the devices shown.

[0210] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0211] like Figure 9 As shown, the computing device cluster includes at least one computing device 1500. The memory 1520 of one or more computing devices 1500 in the computing device cluster may store the same instructions for performing the methods described above.

[0212] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also each store a portion of the instructions for executing the above-described methods. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions of the above-described methods.

[0213] It should be noted that the memory 1520 in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1520 of different computing devices 1500 can implement the functions of one or more modules within the aforementioned device.

[0214] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 One possible implementation is shown. For example... Figure 10 As shown, the two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0215] It should be understood that Figure 10 The functions of computing device 1500A shown can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0216] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0217] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method described above.

[0218] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0219] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0220] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0224] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a model, characterized in that, The method includes: Collect information from the first model, which includes the execution time required for each of the multiple operators in the first model. A recomputation strategy is formulated based on the communication time of the first model during the communication phase and the execution time required by each of the multiple operators in the first model. The recomputation strategy includes: at least one recomputation operator among the multiple operators, and the timing of the at least one recomputation operator executing the recomputation process. The recomputation operator is the operator among the multiple operators used to execute the recomputation process. The timing of the at least one recomputation operator executing the recomputation process includes: executing the recomputation process in parallel with the communication phase of the first model. The first model is trained according to the recomputation strategy.

2. The method according to claim 1, characterized in that, The information of the first model also includes the dependencies between the multiple operators, including the first computation operator depending on the second computation operator. When the output of the second computation operator is already stored in the video memory, the first computation operator performs the recomputation process based on the output of the second computation operator.

3. The method according to claim 1 or 2, characterized in that, The at least one recomputation operator includes at least one target recomputation operator that executes the recomputation process in parallel with the first communication phase, wherein the execution time required for the at least one target recomputation operator to execute the recomputation process is less than or equal to the communication time of the first communication phase.

4. The method according to any one of claims 1 to 3, characterized in that, The information of the first model also includes the data size output by each of the plurality of operators, wherein the sum of the video memory capacity occupied by the data output by the at least one recalculation operator in the plurality of operators during the recalculation process and the video memory capacity occupied by the data in the resident video memory is less than or equal to the total capacity of the video memory.

5. The method according to any one of claims 1 to 4, characterized in that, The information of the first model also includes the type of each operator, wherein the type of the operator includes a communication operator and a computation operator, and the at least one recomputation operator does not include the communication operator.

6. The method according to any one of claims 1 to 5, characterized in that, The step of formulating a recalculation strategy based on the communication time of the first model during the communication phase and the information of the first model includes: Construct an objective function, which is the total execution time required for the at least one recomputation operator to perform the recomputation process, and the independent variables of the objective function include information from the first model; The dependent variable of the objective function is determined based on the objective task, wherein the dependent variable of the objective function is used to indicate the recomputation strategy, and the objective task is to minimize the total execution time.

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, and each cloud data center having at least one server.

8. A device for model training, characterized in that, The device includes: The information acquisition module is used to acquire information about the first model, which includes the execution time required for each of the multiple operators in the first model. A strategy formulation module is used to formulate a recomputation strategy based on the communication time of the first model in the communication phase and the execution time required by each of the multiple operators in the first model. The recomputation strategy includes: at least one recomputation operator among the multiple operators, and the timing of the at least one recomputation operator executing the recomputation process. The recomputation operator is an operator among the multiple operators used to execute the recomputation process, and the timing of the at least one recomputation operator executing the recomputation process includes: executing the recomputation process in parallel with the communication phase of the first model. The strategy execution module is used to train the first model according to the recomputation strategy.

9. The apparatus according to claim 8, characterized in that, The information of the first model also includes the dependencies between the multiple operators, including the first computation operator depending on the second computation operator. When the output of the second computation operator is already stored in the video memory, the first computation operator performs the recomputation process based on the output of the second computation operator.

10. The apparatus according to claim 8 or 9, characterized in that, The at least one recomputation operator includes at least one target recomputation operator that executes the recomputation process in parallel with the first communication phase, wherein the execution time required for the at least one target recomputation operator to execute the recomputation process is less than or equal to the communication time of the first communication phase.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The information of the first model also includes the data size output by each of the plurality of operators, wherein the sum of the video memory capacity occupied by the data output by the at least one recalculation operator in the plurality of operators during the recalculation process and the video memory capacity occupied by the data in the resident video memory is less than or equal to the total capacity of the video memory.

12. The apparatus according to any one of claims 8 to 11, characterized in that, The information of the first model also includes the type of each operator, wherein the type of the operator includes a communication operator and a computation operator, and the at least one recomputation operator does not include the communication operator.

13. The apparatus according to any one of claims 8 to 12, characterized in that, The strategy formulation module is specifically used for: Construct an objective function, which is the total execution time required for the at least one recomputation operator to perform the recomputation process, and the independent variables of the objective function include information from the first model; The dependent variable of the objective function is determined based on the objective task, wherein the dependent variable of the objective function is used to indicate the recomputation strategy, and the objective task is to minimize the total execution time.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The device is applied to a cloud management platform for managing infrastructure that provides cloud services, the infrastructure including at least one cloud data center, and each cloud data center having at least one server.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 7.

16. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 7.

Images