Deep learning optimizer encapsulation method

Abstract

This invention provides a method for encapsulating deep learning optimizers, comprising: encapsulating optimizers; encapsulating optimizers from different deep learning backends and different hardware into a common interface to reduce the learning cost for users; and selecting the optimal optimizer function based on the characteristics of different deep learning backends to reduce the learning cost of using specific optimization functions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for encapsulating a deep learning optimizer. Background Technology

[0002] Deep learning tasks typically use optimizers to optimize the parameters of neural networks. The optimization process generally consists of three steps: forward inference of the neural network, backpropagation of gradients, parameter optimization, and gradient zeroing.

[0003] Different deep learning tasks may employ different optimization features. For example, mixed-precision training may be used to reduce memory usage and improve training speed; gradient accumulation algorithms may be used to reduce memory requirements. Users typically need to implement these training features themselves based on the deep learning backend they are using.

[0004] Different deep learning backends (PyTorch, TensorFlow, PaddlePaddle, etc.) have different optimizer interfaces and different scheduling logics; different hardware platforms (GPU, TPU, IPU, etc.) also have different limitations in the optimization process of neural networks when connecting to different deep learning backends; therefore, users face a large learning cost when implementing corresponding optimization functions based on specific backends and specific hardware. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning optimizer encapsulation method to solve the problem that existing deep learning methods, which rely on specific backends and hardware to implement corresponding optimization functions, have high learning costs.

[0006] To address the aforementioned technical problems, this invention provides a method for encapsulating a deep learning optimizer, comprising:

[0007] Perform optimizer encapsulation; encapsulate multiple deep learning backends and optimizer class instances of multiple hardware into their own attributes, provide a common interface to reduce the learning cost for users; and design the inheritance hierarchy of different backend optimizer encapsulations based on the characteristics of deep learning backends, under the premise of a unified interface.

[0008] Furthermore, based on a unified interface, optimization strategies for different backends are implemented to reduce the learning cost for users to adopt these optimization strategies.

[0009] Based on the characteristics of different deep learning backends, different inheritance levels are selected through optimizer encapsulation to reduce the learning cost of using specific optimization strategies. In other words, based on the characteristics of different deep learning backends, optimizer functionality is encapsulated into unified parameters or abstracted into new optimizer encapsulation instances with the same interface, thereby reducing the learning cost for users.

[0010] The optimizer functionality is selected by the user, and the optimizer encapsulation is designed with different inheritance levels based on the characteristics of different deep learning backends. For example... Figure 3 Inheritance hierarchy in the context of inheritance.

[0011] Optionally, the deep learning optimizer encapsulation method further includes:

[0012] Abstract (extract) the optimizer encapsulation base class to provide a generic optimizer interface;

[0013] The optimizers of different deep learning backends are encapsulated to form optimizer encapsulation subclasses, which can perform gradient backpropagation, parameter update, gradient zeroing and gradient accumulation functions.

[0014] Based on the characteristics of the deep learning training backend, a corresponding optimizer encapsulation subclass for mixed-precision training of the backend is derived;

[0015] Supported optimizer strategies include mixed-precision training and gradient accumulation.

[0016] Optionally, the deep learning optimizer encapsulation method further includes:

[0017] Provides a common interface for optimizers of various deep learning backends;

[0018] The convention is based on the parameter update process of the optimizer encapsulation standard; and

[0019] Multiple optimizers are managed using an optimizer encapsulation dictionary, which calls a common interface to return the status information of all optimizers, including learning rate and momentum.

[0020] Optionally, in the deep learning optimizer encapsulation method described above, the general interface includes:

[0021] The parameter update interface is configured to complete a standard gradient backpropagation, parameter update, and gradient zeroing process; when the parameter update interface is called, gradient accumulation is performed according to the optimizer's encapsulated configuration; and

[0022] The optimizer encapsulates the context interface and is configured to enable mixed-precision training and accelerate gradient accumulation.

[0023] Optionally, in the deep learning optimizer encapsulation method, the parameter update process based on the optimizer encapsulation standard includes:

[0024] Enable the optimizer's encapsulated context interface to perform forward inference of the network within the context, achieving mixed-precision training or accelerating gradient accumulation; and

[0025] Call the parameter update interface outside the context of the optimizer's encapsulated context interface, and update the parameters using the gradient accumulation and gradient truncation functions built into the parameter update interface.

[0026] Optionally, in the deep learning optimizer encapsulation method, the optimizer encapsulation base class includes a base class interface, which includes:

[0027] The interface includes: parameter update interface, gradient backpropagation interface, stepping interface, gradient zeroing interface, get state dictionary interface, load state dictionary interface, optimizer encapsulation context interface, iteration count initialization interface, get learning rate interface, get momentum interface, and get parameter set interface.

[0028] Optionally, in the deep learning optimizer encapsulation method, the optimizer encapsulation subclass has the same interface as the base class, encapsulates optimizers for different deep learning backends, and implements the same interface as the base class optimizer encapsulation.

[0029] The optimizer encapsulation subclass functionality includes:

[0030] Enabling mixed-precision training involves different procedures depending on the deep learning backend. Therefore, the method for enabling mixed-precision training should be selected based on the specific backend used.

[0031] It supports gradient accumulation and optimizes the efficiency of gradient accumulation during distributed training, accelerating gradient accumulation under distributed training.

[0032] The methods for enabling mixed-precision training based on different backends include:

[0033] It can enable mixed-precision training by initializing parameters;

[0034] It can enable mixed-precision training through context;

[0035] Optionally, the deep learning optimizer encapsulation method further includes:

[0036] Implementing the same interfaces as the base class optimizer encapsulation includes: subclasses implementing parameter updates, gradient backpropagation, stepping, gradient zeroing, loading the state dictionary, obtaining the parameter set, obtaining the learning rate, and obtaining the momentum based on the corresponding backend;

[0037] Enabling mixed-precision training includes: if the corresponding backend can enable mixed-precision training by initializing global variables, there is no need to derive an additional mixed-precision subclass; if it is necessary to overload more interfaces than the threshold, then further derive the corresponding backend subclass to implement mixed-precision training; at the same time, enable the context of the optimizer encapsulation context interface before the network forward inference to ensure that the way each backend optimizer encapsulates and enables mixed-precision training is consistent.

[0038] The gradient accumulation function is implemented by encapsulating the base class with an optimizer.

[0039] By default, without gradient accumulation enabled, gradient synchronization occurs with each parameter update during distributed training. Therefore, the optimizer subclass needs to optimize the gradient synchronization logic based on backend characteristics to avoid unnecessary gradient synchronization during gradient accumulation, thereby accelerating the gradient accumulation process.

[0040] Accelerating gradient accumulation requires enabling the optimizer encapsulation context during model forward derivation.

[0041] Optionally, the deep learning optimizer encapsulation method further includes:

[0042] When training multi-optimizer tasks, such as generative adversarial networks, an optimizer wrapper dictionary is extracted to organize multiple optimizer wrappers, so as to manage multiple optimizer wrappers with the optimizer wrapper dictionary.

[0043] The optimizer encapsulation dictionary includes: gradient zeroing interface, parameter set retrieval interface, state dictionary retrieval interface, state dictionary loading interface, learning rate retrieval interface, and momentum retrieval interface.

[0044] Optionally, in the deep learning optimizer encapsulation method described above,

[0045] The optimizer is encapsulated to provide a universal interface for gradient backpropagation, parameter update, and gradient zeroing, shielding the differences between different deep learning backends and hardware; it also provides a universal interface for mixed-precision training and gradient accumulation.

[0046] The optimizer encapsulation base class derives optimizer encapsulation subclasses for different deep learning backends and different hardware. The subclass interface is the same as the parent class, and implements the corresponding backend functions, hardware gradient backpropagation function, parameter update function, and gradient zeroing function.

[0047] The optimizer encapsulation subclass provides a general interface to implement gradient accumulation functionality, provided that the hardware platform and deep learning backend allow it.

[0048] The optimizer encapsulation selectively derives a subclass of the corresponding backend mixed-precision training based on the characteristics of the corresponding deep learning backend. The subclass interface is consistent with the parent class, and mixed-precision training is enabled when the process is called.

[0049] It provides combinatorial optimizer wrappers, which store multiple optimizer wrappers in key-value pairs, and can save and export the state of multiple optimizers at the same time.

[0050] This invention provides a method for encapsulating deep learning optimizers. By proposing the concept of optimizer encapsulation, it reduces the learning cost for users to implement specific optimization strategies. Optimizer encapsulation packages optimizers from different backends and hardware into a universal interface, further reducing the learning cost for users. Based on the characteristics of different deep learning backends, optimizer encapsulation implements commonly used, high-performance optimizer functions, such as mixed-precision training and gradient accumulation. Users only need simple configuration and interface calls to use training strategies such as mixed-precision training and gradient accumulation.

[0051] The optimizer encapsulation provides a unified interface for optimizers across different backend deep learning frameworks and integrates common optimization strategies such as gradient accumulation and mixed-precision training. Taking training generative adversarial networks (GANs) as an example, users of PyTorch, TensorFlow, and Jax typically need to understand the syntax of their respective backend optimizers and then implement training strategies like gradient accumulation and mixed-precision training. This approach has a high learning curve, and the implementations of these training strategies are often not optimal, leading to lower training efficiency. With the optimizer encapsulation, users of any deep learning backend can use the same interface to optimize models without needing to understand the specific optimizer syntax of their backend. Furthermore, the optimizer encapsulation provides efficient gradient accumulation and mixed-precision training strategies, eliminating the need for users to implement them manually, thus improving both development and training efficiency. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the optimizer packaging and update process according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the optimizer encapsulation base class according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of an optimizer encapsulation subclass according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of an optimizer encapsulation dictionary according to an embodiment of the present invention;

[0056] Figure 5 A computer system is a system and / or method according to the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0058] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0059] In this invention, unless otherwise specified, "arranged on," "arranged above," and "arranged on" do not exclude the possibility of an intermediate element between them. Furthermore, "arranged on or above" merely indicates the relative positional relationship between two components, and in certain cases, such as when the product orientation is reversed, it can also be converted to "arranged below or under," and vice versa.

[0060] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0061] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0062] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0063] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0064] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0065] The deep learning optimizer encapsulation method proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0066] The purpose of this invention is to provide a method for encapsulating a deep learning optimizer to solve the problem that existing deep learning optimization strategies based on specific backends and hardware have high learning costs.

[0067] To achieve the above objectives, the present invention provides a deep learning optimizer encapsulation method, comprising: encapsulating the optimizer; encapsulating optimizers from different deep learning backends and different hardware into a universal interface to reduce the learning cost for users; and designing an inheritance hierarchy for the optimizer encapsulation based on the characteristics of different deep learning backends to implement the basic functions of gradient accumulation and mixed precision training for the optimizer encapsulation, thereby reducing the learning cost of using the corresponding optimization strategies.

[0068] Figures 1-5 The first embodiment of the present invention is provided, which illustrates a schematic diagram of a deep learning optimizer encapsulation method.

[0069] To reduce the learning cost for users implementing specific optimization strategies, this invention proposes the concept of optimizer encapsulation. Optimizer encapsulation packages optimizers from different backends and hardware into a unified (universal) interface, reducing the learning cost for users. Based on the characteristics of different deep learning backends, optimizer encapsulation implements commonly used, high-performance optimizer functions, such as mixed-precision training and gradient accumulation. Users only need simple configuration and interface calls to use training strategies such as mixed-precision training and gradient accumulation.

[0070] This invention provides a unified interface by abstracting an optimizer encapsulation base class, encapsulating optimizers for different backends, and implementing functions such as gradient backpropagation, parameter optimization, gradient zeroing, and gradient accumulation. Furthermore, based on the characteristics of the training backend, subclasses are derived for corresponding backend mixed-precision training. For example... Figure 1 As shown, after starting, the optimizer encapsulation context interface is enabled to update parameters. The parameter update includes: gradient calculation, determining whether to update parameters, stepping if yes, and clearing the gradient to zero; otherwise, it ends directly.

[0071] like Figure 2 As shown, the optimizer encapsulation base class includes base class interfaces, which are designed as follows: parameter update interface, gradient backpropagation interface, stepping interface, gradient zeroing interface, get state dictionary interface, load state dictionary interface, optimizer encapsulation context interface, iteration count initialization interface, get learning rate interface, get momentum interface, and get parameter set interface.

[0072] The parameter update interface is named `update_params`. Its function is described as: accepting a `loss` parameter for calculating gradients and updating parameters. `update_params` performs gradient calculation, parameter update, and gradient clearing in sequence. Additionally, `update_params` uses an internal counter; each call increments `_inner_count`, thus accumulating gradients. The interface's parameter list includes the `loss` parameter used to calculate the gradients; other parameters vary depending on the training backend.

[0073] The gradient backpropagation interface is named `backward`. Its function is described as providing a unified gradient calculation encapsulation for optimizations across different precisions and platforms. By default, each call to `backward` increments the internal counter. The interface's parameter list includes the `loss` parameter used to calculate the gradient; other parameters adjust depending on the training backend.

[0074] The step interface, named "step," provides a unified parameter update encapsulation for optimizers with different precision levels and platforms. Its parameter list includes parameters that adjust for different training backends.

[0075] The gradient zeroing interface is named zero_grad. Its function description includes: zeroing the gradients of all managed parameters of the optimizer. The parameter list of this interface includes: adjusting according to different training backends.

[0076] The interface for obtaining the state dictionary is named `state_dict`. Its function description includes: providing a unified encapsulation for exporting the optimizer state dictionary for optimizers of different precisions and platforms. The parameter list for this interface includes parameters that adjust according to different training backends.

[0077] The interface for loading the state dictionary is named `load_state_dict`. Its functionality includes providing a unified wrapper for loading the optimizer state dictionary for optimizers of different precisions and platforms. The parameter list for this interface is adjustable depending on the training backend.

[0078] The optimizer encapsulation context interface is named `optim_context`. This interface provides a unified interface for optimizers of different precisions and platforms, enabling contexts related to mixed-precision training and gradient accumulation acceleration. The base class returns an empty context by default. The parameter list for this interface includes: the model to be optimized.

[0079] The interface for initializing iteration counts is named `initilize_iter_status`. Its function is described as: initializing the internal counter and calculating gradient accumulation parameters based on the maximum number of calls. This interface is independent of the deep learning backend. The parameter list for this interface includes: the optimized training model, the initial number of iterations (`init_counts`), and the maximum number of training iterations (`max_counts`).

[0080] The interface for obtaining the learning rate is named get_lr. The function description of this interface includes: providing a unified interface for obtaining the optimizer learning rate of different training backends. There is no parameter list for this interface.

[0081] The interface for obtaining momentum is named get_momentum. The function description of this interface includes: providing a unified interface for obtaining optimizer momentum from different training backends. There is no parameter list for this interface.

[0082] The interface name for retrieving parameter groups is params. The function description of this interface includes: returning all parameters managed by the optimizer. There is no parameter list for this interface.

[0083] like Figure 3 As shown, the optimizer encapsulation subclass has the same interface as the base class and needs to implement the following three functions: First, encapsulate optimizers for different deep learning backends, implementing the same interface as the base class optimizer encapsulation. Second, enable mixed-precision training; the process for enabling mixed-precision training differs for different deep learning backends. For example, Pyorch's amp requires enabling the autocast context manager, while TensorFlow requires setting a global optimization strategy during initialization. Therefore, different backends enable mixed-precision training in different ways, allowing both enabling mixed-precision training through initialization parameters and enabling it through the context. If needed, the optimizer encapsulation supporting mixed-precision training can be implemented separately. Finally, support gradient accumulation and implement its distributed optimization.

[0084] For the first function of the interface of the optimizer encapsulation subclass, the subclass needs to implement update_params, backward, step, zero_grad, load_state_dict, param_groups, get_lr, and get_momentum based on the corresponding backend.

[0085] For the second function of the optimizer encapsulation subclass interface, if the corresponding backend can enable mixed-precision training by initializing global variables, there is no need to derive an additional mixed-precision subclass. If more interfaces need to be overridden, the corresponding backend subclass can be further derived to implement mixed-precision training. To ensure that the way mixed-precision training is enabled in each backend optimizer encapsulation is consistent, the `optim_context` context needs to be enabled before the network forward inference.

[0086] Regarding the third function of the optimizer wrapper subclass interface, the optimizer wrapper base class has already implemented the gradient accumulation function. However, by default (with gradient accumulation disabled), update_params needs to synchronize the model gradient every time. Therefore, the subclass optimizer wrapper needs to avoid unnecessary gradient synchronization during gradient accumulation based on the characteristics of the corresponding backend.

[0087] Since this feature is an additional function used to accelerate gradient accumulation training in a distributed environment, it is enabled through the optim_context interface, just like mixed-progress training.

[0088] like Figure 4 As shown, this also includes forming an optimizer encapsulation dictionary. When training a generative adversarial network, the generator and discriminator usually require independent optimizers. Therefore, the concept of an optimizer encapsulation dictionary is abstracted to organize multiple optimizer encapsulations. Users can access the optimizer encapsulations stored in the optimizer encapsulation dictionary just like accessing a dictionary.

[0089] Since the loss function is typically wrapped for a single optimizer, while the optimizer wrapper dictionary is a combination of multiple optimizer wrappers, the optimizer wrapper dictionary only serves as a container for optimizer wrappers and does not implement methods such as update_params and backward.

[0090] The optimizer encapsulation dictionary includes: gradient zeroing interface, parameter set retrieval interface, state dictionary retrieval interface, state dictionary loading interface, learning rate retrieval interface, and momentum retrieval interface.

[0091] Gradient zeroing interface name: zero_grad; Interface function description: Clears all gradients encapsulated by the optimizer. The parameter list of this interface includes: adjusts according to different training backends.

[0092] The interface name for retrieving parameter groups is: params_group; the interface function description is: return all parameter groups encapsulated by the optimizer in the form of a dictionary; the parameter list of this interface includes: none.

[0093] The interface name for obtaining the state dictionary is `state_dict`. The interface function description is: it returns a dictionary containing all parameter sets encapsulated by the optimizer; the parameter list for this interface includes parameters that adjust depending on the training backend.

[0094] The interface name for loading the state dictionary is: load_state_dict; the interface function description is: load the state dictionary of all optimizers in dictionary form; the parameter list of this interface includes: adjustable according to different training backends.

[0095] The interface name for obtaining the learning rate is get_lr; the interface function description is: retrieve the learning rate of all optimizers in dictionary form; the parameter list of this interface includes: none.

[0096] The interface name for obtaining momentum is get_momentum; the interface function description is: obtain the momentum of all optimizers in the form of a dictionary; the parameter list of this interface includes: none.

[0097] This invention provides a unified interface for optimizers in various deep learning backends, with the core protected interfaces being the parameter update interface and the optimizer encapsulation context interface. This invention defines a parameter update process based on the optimizer encapsulation standard. This invention uses an optimizer encapsulation dictionary to manage multiple optimizers and can call a unified interface to return the learning rate, momentum, and other status information of all optimizers.

[0098] The parameter update interface `update_params` completes a standard gradient backpropagation, parameter update, and gradient zeroing process. When this interface is called, gradient accumulation is performed according to the configuration encapsulated by the optimizer; the optimizer context interface `optim_context` enables mixed-precision training and accelerates gradient accumulation.

[0099] The parameter update process based on the optimizer encapsulation standard includes: enabling the `optim_context` context, performing forward inference of the network within the context to achieve mixed-precision training or accelerate gradient accumulation; and calling `update_params` outside the `optim_context` context to update the parameters (which includes built-in gradient accumulation and gradient truncation functions).

[0100] The optimizer encapsulation of this invention aims to provide users with a unified interface for gradient backpropagation, parameter optimization, and gradient zeroing, shielding them from the differences between various deep learning backends and training hardware; it provides users with a unified interface for using mixed-precision training, gradient accumulation, and other common optimization strategies, offering the following advantages:

[0101] 1. The optimizer encapsulation base class derives optimizer encapsulation subclasses for different deep learning backends and different hardware. The subclass interface is the same as the parent class, and implements the basic functions of backpropagation of gradients, parameter optimization, and gradient zeroing for the corresponding backend and hardware.

[0102] 2. The optimizer encapsulation subclass provides a unified interface to implement gradient accumulation, provided that the hardware platform and deep learning backend allow it.

[0103] 3. The optimizer encapsulation selectively derives a subclass for mixed-precision training based on the characteristics of the corresponding deep learning backend. The subclass interface is consistent with the parent class, and mixed-precision training will be enabled when the process is called.

[0104] 4. Provides combinatorial optimizer wrappers. Combinatorial optimizer wrappers store multiple optimizer wrappers in key-value pairs, allowing the state of multiple optimizers to be saved and exported simultaneously.

[0105] The deep learning optimizer encapsulation method derived from the technical solution of this invention can be used in the fields of neural networks, computers, and communications to achieve the following technical effects: It reduces the learning cost for users to implement specific optimization strategies. The principle is that optimizer encapsulation packages optimizers from different backends and hardware into a universal interface, reducing the user's learning cost. Based on the characteristics of different deep learning backends, optimizer encapsulation implements commonly used, high-performance optimizer functions, such as mixed-precision training and gradient accumulation. Users only need simple configuration and interface calls to use training strategies such as mixed-precision training and gradient accumulation.

[0106] Figure 5 A computer system 100 implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention can be implemented in... Figure 5 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0107] like Figure 5 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 5Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes a hard disk drive interface 109 for reading and writing to hard disk 110, an optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. Drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0108] When the present invention is Figure 5 When implemented on the computer system 100, the concept of optimizer encapsulation reduces the learning cost for users to implement specific optimization strategies. Optimizer encapsulation packages optimizers from different backends and hardware into a universal interface, further reducing the learning cost for users. Based on the characteristics of different deep learning backends, optimizer encapsulation implements commonly used, high-performance optimizer functions, such as mixed-precision training and gradient accumulation. Users only need simple configuration and interface calls to use training strategies such as mixed-precision training and gradient accumulation.

[0109] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0110] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0111] In summary, the above embodiments have provided detailed descriptions of different configurations for deep learning optimizer encapsulation methods. Of course, this invention includes, but is not limited to, the configurations listed in the above embodiments. Any modifications made based on the configurations provided in the above embodiments are within the scope of protection of this invention. Those skilled in the art can apply the principles described in the above embodiments to other similar applications.

[0112] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0113] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A deep learning optimizer packaging method, characterized in that, The method comprises the following steps: carrying out an optimizer package; packaging multiple deep learning backends and multiple hardware optimizers into a general interface through the optimizer package; and designing an inheritance hierarchy of different backend optimizer packages under the premise of a unified interface according to the characteristics of the multiple deep learning backends; wherein the optimizer package subclass is consistent with the base class interface, and the optimizer package subclass is used to package optimizers of different deep learning backends to realize the same interface as the base class optimizer package; the functions of the optimizer package subclass include: starting mixed precision training, since the procedures of starting mixed precision training are different for different deep learning backends, the way of starting mixed precision training is selected according to different backends; and supporting gradient accumulation and optimizing the efficiency of gradient accumulation in distributed training to accelerate gradient accumulation in distributed training; wherein the way of starting mixed precision training according to different backends includes: being able to start mixed precision training through initialization parameters; and being able to start mixed precision training through context; which further includes: realizing the same interface as the base class optimizer package includes: the subclass realizes parameter update, optimizer package context, gradient backpropagation, step, gradient zeroing, loading state dictionary, obtaining parameter group, obtaining learning rate, and obtaining momentum based on the corresponding backend; starting mixed precision training includes: if the corresponding backend can start mixed precision training through initialization of global variables, there is no need to derive an additional mixed precision subclass, if the number of interfaces that need to be overloaded is more than a threshold, further derive a subclass corresponding to the backend to realize mixed precision training; and the context of the optimizer package context interface is started before network forward reasoning to ensure that the way of starting mixed precision training by the optimizer package of each backend is consistent; gradient accumulation is realized through the optimizer package base class; in the default case of not starting gradient accumulation, gradient synchronization will be performed every time the parameter is updated in distributed training, therefore, the optimizer package subclass needs to optimize the gradient synchronization logic according to the characteristics of the backend to avoid unnecessary gradient synchronization when gradient accumulation occurs, so as to accelerate the process of gradient accumulation; accelerating gradient accumulation requires starting the optimizer package context when the model is forward derived.

2. The deep learning optimizer packaging method of claim 1, wherein, Further includes: extracting the optimizer package base class to provide a general optimizer interface; packaging optimizers of different deep learning backends to realize gradient backpropagation, parameter update, gradient zeroing, and gradient accumulation; deriving an optimizer package subclass corresponding to the mixed precision training of the backend according to the characteristics of the deep learning training backend; the supported optimizer strategies include mixed precision training and gradient accumulation.

3. The deep learning optimizer packaging method of claim 2, wherein, Further includes: providing a general interface for the optimizer of each deep learning backend; agreeing on a parameter update procedure based on the optimizer package standard; and using an optimizer package dictionary to manage multiple optimizers to call the general interface to return the state information of all optimizers, the state information including learning rate and momentum. The general interface includes:

4. The deep learning optimizer packaging method of claim 3, wherein, a parameter update interface configured to complete a standard gradient backpropagation, parameter update, and gradient zeroing procedure; when the parameter update interface is called, gradient accumulation is performed according to the configuration of the optimizer package; and ​ The optimizer package context interface is configured to start mixed precision training and accelerate gradient accumulation.

5. The deep learning optimizer packaging method of claim 4, wherein, The parameter update process based on the optimizer package standard includes: Starting the optimizer package context interface, performing forward inference of the network in the context, and implementing mixed precision training or accelerating gradient accumulation; and Calling the parameter update interface outside the context of the optimizer package context interface to update the parameters through the gradient accumulation function and gradient truncation function provided by the optimizer package.

6. The deep learning optimizer packaging method of claim 5, wherein, The optimizer package base class includes a base class interface, which includes: The parameter update interface, the gradient backpropagation interface, the step interface, the gradient zeroing interface, the state dictionary acquisition interface, the state dictionary loading interface, the optimizer package context interface, the iteration number initialization interface, the learning rate acquisition interface, the momentum acquisition interface, and the parameter group acquisition interface.

7. The deep learning optimizer packaging method of claim 6, wherein, Further including: When training a multi-optimizer task and generating a generative adversarial network, an optimizer package dictionary is extracted to organize multiple optimizer packages, and the multiple optimizer packages are managed using the optimizer package dictionary; The optimizer package dictionary includes: the gradient zeroing interface, the parameter group acquisition interface, the state dictionary acquisition interface, the state dictionary loading interface, the learning rate acquisition interface, and the momentum acquisition interface.

8. The deep learning optimizer package method of claim 6, wherein The optimizer package is performed to provide a universal gradient backpropagation interface, a parameter update interface, and a gradient zeroing interface, and to shield differences between different deep learning backends and different hardware; and a universal interface is provided for mixed precision training and gradient accumulation; The optimizer package base class derives different deep learning backend and hardware optimizer package subclasses, the subclass interface is unified with the parent class, and the corresponding backend function, hardware gradient backpropagation function, parameter update function, and gradient zeroing function are implemented; The optimizer package subclass provides a universal interface to implement the gradient accumulation function if allowed by the hardware platform and the deep learning backend; The optimizer package selectively derives a subclass for mixed precision training of the corresponding deep learning backend according to the characteristics of the corresponding deep learning backend, the subclass interface is unified with the parent class, and when the process is called, the mixed precision training is started; A combined optimizer package is provided, the combined optimizer package stores multiple optimizer packages in the form of key-value pairs, and can save and export the states of multiple optimizers simultaneously.

Images