Method and apparatus for machine learning model

Abstract

The present disclosure provides a computer-implemented method for a machine learning model, comprising: providing a set of inputs to the machine learning model; obtaining a gradient based on the set of inputs; retrieving a first optimizer state and a second optimizer state; updating the first optimizer state and the second optimizer state based on the gradient; and compressing the updated first optimizer state and the second optimizer state in a separate manner.

Description

TECHNICAL FIELD

[0001] Aspects of the disclosure generally relate to artificial intelligence (AI), and more particularly, to methods and apparatuses for memory-efficient optimizers for training machine learning models. BACKGROUND

[0002] Recently, large-scale models with a large number of parameters have shown impressive small-sample learning capabilities on general tasks such as natural language processing, image classification, etc. Despite their power, training these large models is challenging. Memory capacity is one of the main bottlenecks for training large-scale models. Modern machine learning models such as neural networks are typically trained with stateful optimizers such as Adam, which require one or two optimizer states to be maintained for each parameter, introducing a significant memory overhead that limits both the size of the model used and the number of samples in the mini-batch, both of which have a significant impact on the accuracy of the model.

[0003] Accordingly, it can be desirable to provide a method or technique for improving memory usage of a stateful optimizer. SUMMARY

[0004] The following presents a simplified summary of one or more aspects in accordance with the present disclosure to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key or critical elements of all aspects nor does it delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0005] In an aspect of the disclosure, a computer-implemented method for a machine learning model is provided, comprising: providing a set of inputs to the machine learning model; obtaining a gradient based on the set of inputs; retrieving a first optimizer state and a second optimizer state; updating the first optimizer state and the second optimizer state based on the gradient; and compressing the updated first optimizer state and the second optimizer state in a separate manner.

[0006] In another aspect of the disclosure, a computer-implemented method for a machine learning model is provided, comprising: providing a set of inputs to the machine learning model, wherein the set of inputs comprises an image or a text of natural language; obtaining a gradient based on the set of inputs; retrieving a first optimizer state and a second optimizer state from a memory; updating the first optimizer state and the second optimizer state based on the gradient; and compressing the updated first optimizer state and the second optimizer state in separate manners and storing the compressed updated first optimizer state and the second optimizer state into the memory, wherein parameters of the machine learning model are updated based on the updated first optimizer state and the second optimizer state.

[0007] In another aspect of the disclosure, an apparatus for a machine learning model is provided, comprising a memory and at least one processor coupled to the memory. The at least one processor is configured to provide a set of inputs to the machine learning model; obtain a gradient based on the set of inputs; retrieve a first optimizer state and a second optimizer state; update the first optimizer state and the second optimizer state based on the gradient; and compress the updated first optimizer state and the second optimizer state in separate manners.

[0008] In another aspect of the disclosure, a computer program product for a machine learning model is provided, comprising processor executable computer code to: provide a set of inputs to the machine learning model; obtain a gradient based on the set of inputs; retrieve a first optimizer state and a second optimizer state; update the first optimizer state and the second optimizer state based on the gradient; and compress the updated first optimizer state and the second optimizer state in separate manners.

[0009] In another aspect of the disclosure, a computer readable medium stores computer code for a machine learning model. The computer code, when executed by a processor, causes the processor to: provide a set of inputs to the machine learning model; obtain a gradient based on the set of inputs; retrieve a first optimizer state and a second optimizer state; update the first optimizer state and the second optimizer state based on the gradient; and compress the updated first optimizer state and the second optimizer state in separate manners.

[0010] The proposed method for memory efficient optimization of machine learning models according to one or more aspects of the disclosure can be used in various applications and / or fields, such as image classification, property language processing or generation, question answering, machine translation, etc. Although described below for some content with reference to language and vision tasks for illustration purposes only, the disclosure can be applicable to many other application fields beyond these specific application fields, such as graph learning tasks, audio tasks, reinforcement learning, etc.

[0011] Considering the respective characteristics of different optimizer states, memory efficiency can be further improved by using separate compression procedures for different optimizer states, while maintaining comparable accuracy compared to full-precision counterparts.

[0012] Other aspects or variations of the present disclosure, as well as other potential advantages, will become more apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] The disclosed aspects will be described with reference to the accompanying drawings, which are provided for illustration purposes and are not intended to limit the disclosed aspects.

[0014] Figure 1A and 1B A diagram illustrating optimizer state tensors is shown, in accordance with one or more aspects of the present disclosure.

[0015] Figure 2A and Figure 2B A diagram illustrating the magnitude of two first-order momentum tensors in two layers of a Swin-T model during training is shown.

[0016] Figure 3 A diagram illustrating a first-order momentum distribution is shown.

[0017] Figure 4A and Figure 4B A diagram illustrating the distribution of approximations of first-order momentum quantized by block sizes of 2048 and 128, respectively, is shown.

[0018] Figure 5A , 5B and 5C show a diagram illustrating the distribution density of the inverse of the square root of second-order momentum.

[0019] Figure 6 A diagram illustrating an exemplary workflow of a method for compressing optimizer states, in accordance with one or more aspects of the present disclosure, is shown.

[0020] Figure 7 A diagram illustrating an exemplary framework for compressing optimizer states, in accordance with one or more aspects of the present disclosure, is shown.

[0021] Figure 8 A diagram illustrating an example of a hardware implementation for an apparatus, in accordance with one or more aspects of the present disclosure, is shown. DETAILED DESCRIPTION

[0022] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed solely for the purpose of enabling those skilled in the art to better understand and thus implement embodiments of the present disclosure, and not to place any limitations on the scope of the present disclosure.

[0023] Random gradient-based optimization has central practical importance in many scientific and engineering domains. Many problems in these domains can be cast as optimization of some scalar parameterized objective function that needs to be maximized or minimized with respect to its parameters. Often, the objective function is stochastic. For example, many objective functions consist of a sum of sub-functions evaluated at different sub-samples of data; in this case, optimization can be made more efficient by taking gradient steps with respect to individual sub-functions (i.e., stochastic gradient descent (SGD) or ascent). SGD itself proves to be an efficient and effective optimization method that is central to many machine learning success stories, such as recent advances in deep learning.

[0024] The state optimizer maintains gradient statistics over time, e.g., exponential smoothing of past gradient values (e.g., SGD with momentum) or squared sum (e.g., Adam). This state can be used to accelerate optimization compared to vanilla SGD. However, this state also occupies memory that could otherwise be allocated to model parameters, limiting the size of the model.

[0025] There are several approaches to reduce the memory consumption of optimizers. In general, decomposition applies a low-rank approximation to the optimizer state, incremental adjustment avoids maintaining the optimizer state for most parameters by adjusting only a small subset, and low-precision optimizers utilize low-precision numerical formats to represent their state, which consume less memory.

[0026] To further illustrate, the general update rule for the Adam optimizer can be formulated as follows: Equation (1) During the training process, the model parameters and the optimizer state (e.g., first and second moment) , need to be persistently stored in memory (e.g., cache of a GPU). As the model gets larger, the optimizer state is the main source of training memory consumption. For example, each parameter dimension can take two 32-bit optimizer states, making the memory consumption of an Adam-type optimizer three times that of a stateless optimizer such as SGD.

[0027] To mitigate the significantly increased memory consumption of stateful optimizers, a compression-based memory-efficient optimization framework can be used, which is described in Algorithm 1.

[0028] Algorithm can be any gradient-based optimizer, such as SGDM, Adam, AdamW, etc. For example, the exact state only temporarily, and only compressed state is stored persistently in memory. Since in neural networks, the state vector and is usually a concatenation of the state vectors of each parameterized layer, a memory footprint reduction can be achieved. Thus, the optimizer steps of lines 3 to 5 can be performed separately for each layer, such that only the exact state of one single layer is present in memory at a time. The states of all other layers can be kept compressed.

[0029] Recent research has proposed various methods for compression, e.g., Dettmer et al. (“8-bit optimizers via block-wise quantization,” in International Conference on Learning Representations (2022b) propose an 8-bit optimizer with a reparameterized embedding layer (“stable embedding layer”) and a block-wise 8-bit dynamic exponential numerical format for the optimizer state; Shazeer et al. (“Adafactor: Adaptive learning rate method with second order

[0030] Quantizing the optimizer state to a lower precision is an effective way to compress the optimizer state. In this case, the optimizer state is compressed with a quantizer and decompressed with a dequantizer. Low-precision numerical formats can significantly impact the accuracy of the quantization method. Here, a general framework for numerical formats is given as follows.

[0031] A quantizer can convert a full-precision tensor to a low-precision format. Based on the formulas proposed in recent research, a quantizer can be unwound into two parts: a normalization and a mapping which are applied sequentially and element-wise to the tensor to be quantized. Formally, a quantizer for a tensor can be given by Equation (2) The normalization operator scales each element of into the unit interval. Normalization can have different granularities, such as per-tensor, per-token (row), per-channel (column), per group, and block-wise. As an example, the per-tensor and block-wise normalization operators can be given as follows: Equation (3) The scaling factor involved can be referred to as a quantization scale, which is stored persistently along with the quantized tensor until dequantization. The granularity of normalization presents a trade-off between quantization error and memory overhead. A normalization method with low quantization error and acceptable memory overhead can be preferred. In this example, the coarsest per-tensor normalization operator has negligible memory overhead, i.e., only one scaling factor independent of the tensor size, but suffers the largest error. Block-wise normalization treats a tensor as a one-dimensional array, divides the array into blocks of size (e.g., referred to as blocks), and assigns a quantization scale within each block, which results in a total of quantization scales.

[0032] Mapping converts the normalized quantity to a low-bitwidth integer. Formally, the mapping operator is equipped with a bitwidth and a predefined increasing mapping, i.e., quantization mapping . Then can be defined as follows: Equation (4) The current mapping can include a dynamic exponent (DE) mapping. DE mapping can approximate small values well, similar to floating-point numbers. DE splits the binary representation of a low-precision integer into three parts: a leading sequence of zeroes, followed by an indicator bit, and the remaining fraction bits. DE can be defined as: Equation (5) The dequantizer can simply be the inverse of the quantizer, which can simply be: Equation (6) While state optimizers with current compression methods can achieve similar convergence as full-precision optimizers on language modeling, image classification, machine translation, and language understanding tasks, the present disclosure aims to further improve memory efficiency while maintaining good convergence.

[0033] In one or more aspects of the present disclosure, a dedicated compression style or process (e.g., various quantization methods and / or factorization, etc.) can be designed for each of the optimizer states by considering the respective characteristics of the two optimizer states. By adapting to the respective characteristics of each order of momentum, memory efficiency can be further improved while maintaining good convergence.

[0034] For example, one of the important properties of the second-order momentum can be that quantizing a non-zero amount to zero would cause a significant bias during training, since the update direction is typically inversely proportional to the square root of the second-order momentum, as shown in equation (1). On the other hand, quantization of the first-order momentum can be relatively simple. For example, the first-order momentum can be compressed by a lower-precision quantizer (e.g., 4 bits or lower), while the second-order momentum can be compressed by a decomposition or higher-precision quantizer (e.g., 8 bits or higher), which can avoid or help mitigate the zero-point problem.

[0035] In particular, one remedy for the zero-point problem is to simply remove the zeros from the DE quantization graph, which can be referred to as DE-0. The minimum value that can be represented by DE-0 is 0.0033. By using DE-0, the zero-point problem can be avoided, but since the zeros are removed, DE-0 wastes one of the quantization points, which can reduce the quantization precision. Therefore, a higher-precision quantizer with more quantization points can be used to compensate for the wasted points, e.g., an 8-bit DE-0 quantizer with 255 (i.e., 256-1) effective quantization points can be used for the second-order momentum.

[0036] To further improve performance, the patterns of the first-order and second-order momenta are studied, and various improvements are presented herein to make even better memory efficiency. For example, there are some outliers in the momenta. The outliers can significantly affect the quantization scale due to their large magnitudes. We find that the outlier patterns in the first-order and second-order momenta are quite complex.

[0037] Figure 1A and 1B A diagram illustrating an optimizer state tensor is shown. Figure 1A A two-dimensional tensor is shown, where values are defined based on two coordinate axes (x, y) (e.g., block 101). Figure 1B A three-dimensional tensor is shown, where values are defined based on three coordinate axes (x, y, z) (e.g., block 102). In other aspects of the disclosure, the optimizer state tensor can have higher dimensions (e.g., greater than three dimensions) or one dimension (e.g., a vector).

[0038] Figure 2A and Figure 2B A diagram illustrating the magnitudes of two first-order momentum tensors in two layers of a Swin-T model during training is shown. As shown in Figure 2A the outliers of the first layer are located in fixed rows, and as shown in Figure 2B the outliers of the second layer are located in fixed columns, where the tensors are two-dimensional.

[0039] The complex outlier patterns can make the optimizer state more difficult to quantize. For example, there can be some momentum tensors where the outliers roughly stay in certain columns, as shown inFigure 2B Block-wise normalization treats the tensor as a one-dimensional sequence row by row. Thus, when the block size is relatively large, any block can include entries in the outlier column, resulting in a large amount of scale within each block. In this case, block-wise normalization can not be superior to per-tensor normalization.

[0040] Figure 3 A plot showing the distribution of the first-order momentum is shown. From Figure 3 It can be seen that the distribution of the magnitude of the first-order momentum can resemble a normal distribution, and the majority of the quantity of the first-order momentum is concentrated around zero, and the number of outliers is much smaller.

[0041] In one or more aspects of the present disclosure, a smaller block size (e.g., 128) is proposed because it can provide enhanced performance while incurring only a small memory overhead.

[0042] Figure 4A and Figure 4B A plot showing the distribution of the approximation of the first-order momentum quantized by a block size of 2048 and a block size of 128, respectively, is shown. From Figure 4A and Figure 4B It can be seen from the comparison between Figure 3 that quantization with a smaller block size (such as 128) can better approximate the first-order momentum (e.g., as shown in

[0043] It should be appreciated that other smaller block sizes (e.g., 64 or even smaller block sizes, or block sizes of 256, 512, 1024, etc.) can be employed without causing a departure from the scope of the present disclosure, depending on the particular application circumstances (e.g., memory budget, distribution of outliers, model size, etc.).

[0044] Figure 5A , 5B and 5C show plots of the distribution density of the inverse of the square root of the second-order momentum (e.g., transformed with where Figure 5A represents the full-precision second-order momentum, Figure 5B represents the approximation of the second-order momentum quantized by a block size of 128 and DE mapping, and Figure 5C represents the approximation of the second-order momentum quantized by a block size of 128 and DE-0 mapping, and all plots are in the log10 scale. From Figure 5B It can be seen that even with a block size of 128, the performance of the first-order momentum can be significantly improved (e.g., as shown in Figure 4B but the approximation of the second-order momentum can not be as good.

[0045] This is because quantizing the second-order momentum can be more difficult and incur training instability compared to the first-order momentum. For example, the zero-point problem can be a major bottleneck for quantizing the second-order momentum. As shown in Figure 5B , the quantizer pushes most entries of the tensor to zero, and thus the inverse of the square root of most of the points falls into due to the zero-point problem, resulting in the approximation being completely degraded. It can be seen that excluding zero from the mapping is indeed a key factor for the second-order momentum quantization, which cannot be replaced by a smaller block size. Moreover, the second-order momentum can have a sharper outlier pattern and ill-conditioned distribution compared to the first-order momentum.

[0046] While the DE-0 mapping can resolve the zero-point problem as shown in Figure 5C , it comes at the cost of reducing quantization precision due to wasting one of the quantization bits. In this case, we propose to use a linear mapping (e.g., ) with the minimum representable value of 0.0625. For quantizing the second-order momentum, the linear mapping can perform better than DE-0, because it can resolve the zero-point problem without wasting the quantization bit width. For example, a linear mapping with a bit width can be used for the second-order momentum.

[0047] Alternatively, to resolve the zero-point problem, a decomposition method can be applied to the second-order momentum while a quantization method is used for the first-order momentum, because the decomposition method can only be applicable to the second-order momentum.

[0048] Moreover, to meet the challenge of the more complex outlier pattern of the second-order momentum, we propose rank-1 normalization in this paper. Formally, for a non-negative tensor of matrix shape , its one-dimensional statistical information and can be defined as and , which are exactly the per-row and per-column normalized quantization scales. The proposed rank-1 normalization can jointly utilize the two quantization scales, and produces a more stringent bound for an entry, which can be defined as: Equation (7) Rank-1 normalization can utilize one-dimensional information in a more fine-grained way compared to per-tensor, per-token (row), and per-channel (column) normalization, and can provide element-specific quantization scales. The element-specific quantization scales can be intelligently determined based on the one-dimensional information associated with each element. Thus, when the outlier pattern is more complex (such as having an unknown and / or varying distribution), outliers can be more intelligently and effectively handled by using rank-1 normalization.

[0049] Figure 6An example workflow of a method for compressing optimizer states according to one or more aspects of the disclosure is shown. The method 600 can be used to train and / or fine-tune machine learning models for various tasks, e.g., image classification with Swin-T, natural language understanding by fine-tuning RoBERTa-L, question answering by fine-tuning RoBERTa-L, natural language generation by fine-tuning GPT-2Medium, machine translation by training Transformer-Base, etc.

[0050] At step 610, a set of inputs can be provided to a machine learning model. The set of inputs can include a mini-batch of samples, which can be obtained from an image dataset, a text dataset, etc.

[0051] At step 620, gradients can be obtained based on the set of inputs, e.g., as shown in Algorithm 1.

[0052] At step 630, a first optimizer state and a second optimizer state can be retrieved. For example, the first optimizer state and the second optimizer state can have been previously compressed and stored into memory. During the current iteration, the compressed first optimizer state and the second optimizer state can be fetched from memory and decompressed. For another example, the first optimizer state and the second optimizer state can be initial states.

[0053] At step 640, the retrieved decompressed first optimizer state and the second optimizer state can be updated based on the gradients obtained at step 620, e.g., as shown in Algorithm 1, to obtain updated first optimizer state and second optimizer state, which can be used to update parameters of the machine learning model.

[0054] At step 650, the updated first optimizer state and the second optimizer state can be compressed in a separate manner. For example, one or more of a 4-bit quantizer, a higher bit-width quantizer (such as 8-bit or higher), block-wise normalization with a smaller block size (e.g., 64, 128, 256, or 512), rank-1 normalization, block-wise normalization with a large block size (e.g., 2048), per-tensor normalization, per-row normalization, per-column normalization, linear mapping, DE mapping, DE-0 mapping, or decomposition can be determined for compressing the first optimizer state and compressing the second optimizer state, respectively. The determination can be made based on, for example, characteristics of the first optimizer state and the second optimizer state exhibited in the optimization algorithm and / or the model. By using separate compression procedures for different optimizer states, individual characteristics of each of the different optimizer states can be accommodated, and thus memory efficiency can be further improved while maintaining good convergence. For applicable tasks, mixed compression for optimizers can benefit from both aspects: good convergence and memory efficiency.

[0055] In an example, the first optimizer state (i.e., first order momentum) can be compressed using a 4-bit quantizer, e.g., with DE mapping, bit-width of 4, and rank-1 normalization or large block size (e.g., 2048) normalization.

[0056] In an example, the second optimizer state (i.e., second order momentum) can be compressed using an 8-bit quantizer, e.g., with DE-0 mapping, bit-width of 8, and rank-1 normalization or large block size (e.g., 2048) normalization.

[0057] In an example, the second optimizer state (i.e., second order momentum) can be compressed using a 4-bit quantizer, e.g., with linear mapping, bit-width of 4, and rank-1 normalization.

[0058] In an example, the second optimizer state (i.e., second order momentum) can be compressed by decomposing the tensor of the second optimizer state with dimension higher than 1 and quantizing the remaining tensors of the second optimizer state with dimension of 1. In the case of one-dimensional tensors, rank-1 normalization can fall back to per-tensor normalization, thus block-wise normalization (e.g., with small block size of 128, etc.) can be used.

[0059] In one or more aspects of the disclosure, for example, the parameters of the machine learning model can be updated based on the updated first optimizer state and the second optimizer state, as shown in Equation (1). Generally, the above steps can be repeated with smaller batches of samples until convergence. Once the machine learning model converges, it can be deployed in various environments for prediction to control or guide corresponding operations in the particular environment. For example, the updated machine learning model can be provided with data from sensors as input (e.g., images captured by vision sensors) and output data or signals to control actuators or effectors to perform corresponding actions (such as slowing down in an autonomous driving environment).

[0060] It should be appreciated that the above examples are for illustration only, and the methods presented herein can be applied to other specific application areas without causing a departure from the present disclosure.

[0061] Figure 7An exemplary framework for compressing optimizer states is shown in accordance with one or more aspects of the present disclosure. The framework 700 can include a logic unit 710 to hold an optimization algorithm for a model, and a memory 720. For example, the method 600 can be performed in the framework 700. The logic unit 710 can include software, hardware, firmware, data structures, or any combination thereof, and the memory 720 can include one or more storage units, depending on the particular design preference. A set of inputs 701 can be provided to the logic unit 710, and gradients can be obtained by the logic unit 710 according to the optimization algorithm. The current optimizer state can be compressed and stored in the memory 720. To update the parameters of the model based on the gradients, the logic unit 710 can access the memory 720 to obtain the current optimizer state. After updating the parameters and the current optimizer state, the logic unit 710 can compress the updated optimizer state according to one or more aspects of the proposed methods, and store the compressed updated optimizer state into the memory 720 for subsequent use.

[0062] For example, the update can be performed layer by layer. Thus, only the portion of the compressed current optimizer state associated with one single layer can be decoded at a time, and used to update the corresponding parameters associated with that layer, during which the portion of the current optimizer state associated with that layer is first updated based on the gradients (e.g., as shown in equation (1)). After the update, the updated portion of the current optimizer state associated with that layer can be compressed and stored into the memory 720 for the next iteration.

[0063] To further demonstrate the advantages of the present disclosure, experimental results are given in the following table.

[0064] Table 1: Performance on language and vision tasks (metrics: NLU = average accuracy or correlation; CLS = accuracy; NLG = BLEU (bilingual evaluation understudy); QA = Fl; MT = SacreBleu)

[0065] In Table 1, the 4-bit optimizers herein are compared with their full-precision counterparts as well as other memory-efficient optimizers including 8-bit AdamW + (Dettmer et al. in International Conference on Learning Representations (2022b) “8-bit optimizers via block-wise quantization”), where + indicates that the optimizer state for the stable embedding layer is not quantized), Adafactor (Shazeer et al. in International Conference on Machine Learning (2018) “Adafactor: Adaptive Learning Rate and ++ indicates ) and SM3 (Anil et al., “Memory-Efficient Adaptive Optimization,” in Advances in Neural Information Processing Systems (2019)). For the 4-bit optimizers presented herein, there are two versions based on 32-bit AdamW: Method 1 presented herein quantizes the first-order momentum by using block-wise normalization with a block size of 128 and DE mapping, and quantizes the second-order momentum by using rank-1 normalization and linear mapping; Method 2 presented herein quantizes the first-order momentum by using block-wise normalization with a block size of 128 and DE mapping, and decomposes the tensor of the second-order momentum with a dimension greater than one and quantizes the tensor of the second-order momentum with a dimension of 1 by using rank-1 normalization and linear mapping.

[0066] To further evaluate the memory and computational efficiency of the presented 4-bit optimizers, we compare the methods 1 and 2 presented herein with 32-bit AdamW and 8-bit AdamW, as shown in Table 2.

[0067] Table 2

[0068] As shown in Tables 1 and 2, by employing one or more improvements presented herein, the presented methods can provide significantly improved memory savings compared to current compression methods, while maintaining comparable accuracy compared to the full-precision counterparts.

[0069] Figure 8 An example of a hardware implementation for an apparatus 800 is shown in accordance with one or more aspects of the present disclosure. The apparatus 800 for memory-efficient optimization of models can include a memory 810 and at least one processor 820. The processor 820 can be coupled to the memory 810 and be configured to implement methods and / or frameworks 600 and 700 described above by Figure 6 and Figure 7 the processor 820 executing software instructions resident in the memory 810. The processor 820 can be a general purpose processor, or can be implemented as a combination of

[0070] Various operations, models, and networks described in connection with the present disclosure can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. According to embodiments of the present disclosure, a computer program product for memory-efficient optimization of models can include instructions configured to be implemented by a processor to perform the methods and / or frameworks 600 and 700 described above. The computer program product can include a computer-readable storage medium 820 or memory having stored Figure 6 and Figure 7The described methods and / or the processors of frameworks 600 and 700 can execute computer code. According to another embodiment of the present disclosure, a computer- readable medium can store computer code for memory-efficient optimization of models, which, when executed by a processor, can cause the processor to implement the above described methods and / or frameworks 600 and 700. The computer-readable medium contains both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for example, can be properly termed a computer-readable medium. Other embodiments and implementations are within the scope of the present disclosure. Figure 6 and 7 The described methods and / or the processors of frameworks 600 and 700 can execute computer code. According to another embodiment of the present disclosure, a computer- readable medium can store computer code for memory-efficient optimization of models, which, when executed by a processor, can cause the processor to implement the above described methods and / or frameworks 600 and 700. The computer-readable medium contains both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for example, can be properly termed a computer-readable medium. Other embodiments and implementations are within the scope of the present disclosure.

[0071] The foregoing description of the disclosed implementations is provided to enable any person skilled in the art to make or use the various implementations. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other implementations without departing from the scope of the various implementations. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the claims, the principles and the novel features disclosed herein.

Claims

1. A method for computer-based implementation of a machine learning model, comprising: Provide a set of inputs to the machine learning model; The gradient is obtained based on the set of inputs; Retrieve the first optimizer state and the second optimizer state; The first optimizer state and the second optimizer state are updated based on the gradient; as well as The first optimizer state and the second optimizer state are compressed and updated separately.

2. The computer-implemented method according to claim 1, wherein, Compressing the updated first optimizer state and second optimizer state in separate ways includes: The updated first optimizer state is compressed using 4-bit quantization; and The updated second optimizer state is compressed using 8-bit quantization.

3. The computer-implemented method according to claim 1, wherein, Compressing the updated first optimizer state and second optimizer state in separate ways includes: The updated first optimizer state is compressed using block-by-block normalization with a block size of 128; and The updated second optimizer state is compressed by rank-1 normalization.

4. The computer-implemented method according to claim 1, wherein, Compressing the updated first optimizer state and second optimizer state in separate ways includes: The updated first optimizer state is compressed through quantization; and The updated second optimizer state is compressed through decomposition.

5. The computer-implemented method according to claim 4, wherein, Compressing the updated second optimizer state through decomposition also includes: Decompose the tensor of the updated second optimizer state with a dimension greater than one; and The tensor of the updated second optimizer state with a quantization dimension of one.

6. The computer-implemented method according to claim 1, wherein, Compressing the updated first optimizer state and second optimizer state in separate ways includes: The updated first optimizer state is compressed by quantization using a dynamic exponential (DE) mapping; and The updated second optimizer state is compressed by using linear mapping for quantization.

7. The computer-implemented method according to claim 1, wherein, The parameters of the machine learning model are updated based on the updated first optimizer state and the updated second optimizer state.

8. The computer-implemented method according to claim 7, further comprising: Provide input to the updated machine learning model; as well as The output of the updated machine learning model is obtained to control the actuator to perform operations based on the output.

9. A method for computer-based implementation of a machine learning model, comprising: The machine learning model is provided with a set of inputs, wherein the set of inputs includes images or text in natural language; The gradient is obtained based on the set of inputs; Retrieve the first optimizer state and the second optimizer state from memory; Update the first optimizer state and the second optimizer state based on the gradient; and The first optimizer state and the second optimizer state are compressed and updated in separate ways, and the compressed and updated first optimizer state and the second optimizer state are stored in memory. The parameters of the machine learning model are updated based on the updated first optimizer state and the updated second optimizer state.

10. An apparatus for a machine learning model, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the method according to any one of claims 1-7, and to store an updated first optimizer state and a second optimizer state compressed according to any one of claims 1-7 into the memory.

11. A computer program product for machine learning models, comprising: Processor-executable computer code for performing the method according to any one of claims 1-7.

12. A computer-readable medium storing computer code for a machine learning model, said computer code, when executed by a processor, causing the processor to perform the method of any one of claims 1-7.

Images