A low-bit-width and dynamic parameter initialization model training method

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a low-bit-width and dynamic parameter initialization model training method, which comprises: obtaining a parameter scale corresponding to a target growth stage; based on the parameter scale corresponding to the target growth stage, expanding the parameter scale on the basis of a model trained in a previous growth stage of the target growth stage; one or more parameters in the model after the parameter scale is expanded are quantized by using a low bit-width; the model after the parameter scale is expanded is trained, and a first-precision calculation kernel is called to process the operation between an activation matrix and a weight matrix during the training process, so as to obtain a model trained in the target growth stage; wherein when the model trained in the target growth stage is used to execute an inference task, a second-precision calculation kernel is called to process the operation between the activation matrix and the weight matrix. Through the present disclosure, the cost of model training is saved, the limitation on deployment resources is reduced, and the hardware scheduling complexity is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for initializing model training using low bit width and dynamic parameters. Background Technology

[0002] For large language models, existing training methods are generally costly. Furthermore, large language models typically have a large parameter size, resulting in significant memory or GPU memory consumption and limiting deployment resources. While model quantization can reduce memory or GPU memory consumption to some extent, it increases hardware scheduling complexity. Summary of the Invention

[0003] In view of this, this disclosure proposes a method, apparatus, electronic device, storage medium, and computer program product for model training using low bit width and dynamic parameter initialization.

[0004] According to one aspect of this disclosure, a method for initializing a model training process using low bit width and dynamic parameters is provided, the method comprising:

[0005] Obtain the parameter scale corresponding to the target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth.

[0006] Based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the expanded parameter scale model are quantized with low bit width, and the low bit width means that the number of bits is less than the preset value of the bit width.

[0007] The model with the expanded parameter scale is trained, and during the training process, a first-precision computation kernel is called to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision computation kernel is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

[0008] In one possible implementation, expanding the parameter scale based on the model trained in the previous growth stage of the target growth stage includes:

[0009] The model trained in the previous growth stage is expanded in two dimensions by width expansion and depth expansion; wherein, the width expansion is used to increase the number of attention heads and / or the hidden layer dimension in the model; and the depth expansion is used to increase the number of network layers in the model.

[0010] In one possible implementation, the two-dimensional parameter scaling of the model trained in the previous growth stage through width and depth scaling includes:

[0011] When performing the width expansion, the number of new attention heads and the hidden layer dimension are randomly configured in the model trained in the previous growth stage.

[0012] And / or,

[0013] During the depth expansion, based on the distance between the input and output sequences of each network layer in the model trained in the previous growth stage, one or more network layers in the model trained in the previous growth stage are selected for replication.

[0014] In one possible implementation, the method further includes:

[0015] The newly added attention head and the weight parameters in the hidden layer dimension are initialized to any value in a preset set of values, wherein the preset set of values ​​includes three values: -1, 0, and 1.

[0016] And / or,

[0017] The activation value distribution of the activation matrix corresponding to the one or more network layers is optimized through a preset transformation.

[0018] In one possible implementation, the method further includes:

[0019] The weight matrices in the model after the parameter scaling is ternary and / or in the model trained during the target growth stage are stored using ternary representation.

[0020] The quantization precision of each activation matrix is ​​assigned based on the type of each activation matrix in the model after the expansion parameter scale and / or in the model trained in the target growth stage.

[0021] In one possible implementation, the method further includes:

[0022] Perform outlier channel detection on the activation matrix;

[0023] Each detected outlier channel is placed at the end of the activation matrix where the outlier channel is located, resulting in the reordered activation matrix corresponding to the outlier channel. Based on the position of each channel in the reordered activation matrix corresponding to the outlier channel, the position of each channel in the associated weight matrix is ​​adjusted synchronously so that the adjusted weight matrix is ​​adapted to the reordered activation matrix corresponding to the outlier channel.

[0024] Assign different quantization precisions to the outlier channel and the normal value channel in the activation matrix containing the outlier channel.

[0025] In one possible implementation, training the model with the expanded parameter scale includes:

[0026] During backpropagation, the weights of the output layer in the model with the expanded parameter size are directly calculated using the least squares method, and the gradients of the hidden layer in the model with the expanded parameter size are passed through an estimator.

[0027] According to another aspect of this disclosure, a model training apparatus for initializing low bit width and dynamic parameters is provided, the apparatus comprising:

[0028] An acquisition module is used to acquire the parameter scale corresponding to a target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth.

[0029] The extension module is used to expand the parameter scale based on the parameter scale corresponding to the target growth stage and the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the model after expanding the parameter scale are quantized with low bit width, and the low bit width means that the number of bits is less than a preset value.

[0030] The training module is used to train the model after expanding the parameter scale, and during the training process, it calls a first-precision calculation kernel to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision calculation kernel will be called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

[0031] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0032] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0033] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0034] Through various aspects of this disclosure, the parameter scale corresponding to the target growth stage is obtained; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth; based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage; wherein, in the model after expanding the parameter scale, one or more parameters are quantized using a low bit width, and the low bit width means a bit width with a number of bits less than a preset value;

[0035] The model with expanded parameters is trained, and during training, a first-precision computational kernel is called to process the operations between the activation matrix and the weight matrix, resulting in the model trained in the target growth stage. When the model trained in the target growth stage is used for inference tasks, a second-precision computational kernel is called to process the operations between the activation matrix and the weight matrix. The first precision is greater than the second precision. Thus, by fusing growth-based parameter expansion with low-bit-width quantization, memory and GPU memory usage during training are significantly reduced, training efficiency is improved, and the cost of training machine learning models is saved, achieving efficient training of machine learning models. Simultaneously, the use of low-bit-width quantization for parameters in the model greatly reduces the memory and GPU memory occupied by the model, lowering the limitations on deployment resources. Furthermore, the use of a first-precision computational kernel to process the operations between the activation matrix and the weight matrix during the training stage and a second-precision computational kernel to process the operations between the activation matrix and the weight matrix during the inference stage enables adaptive scheduling of computational kernels, reducing hardware scheduling complexity. The trained model can be deployed on different resources as needed; for example, resources with different precision computational kernels and small memory and GPU memory requirements can support model deployment.

[0036] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0037] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0038] Figure 1 The flowchart illustrates a method for training a model using low bit width and dynamic parameter initialization according to an embodiment of the present disclosure.

[0039] Figure 2 A schematic diagram illustrating the training loss of a growth training model according to an embodiment of the present disclosure is shown.

[0040] Figure 3(a) shows a schematic diagram of the model training loss variation curves under different combinations of hyperparameters according to an embodiment of the present disclosure.

[0041] Figure 3(b) shows a schematic diagram of the gradient vector norm variation curves under different combinations of hyperparameters according to an embodiment of the present disclosure.

[0042] Figure 4 A schematic diagram illustrating the effect of low-bit quantization according to an embodiment of the present disclosure is shown.

[0043] Figure 5(a) shows a comparative schematic diagram of the inference speed of different models deployed using a certain chip according to an embodiment of the present disclosure.

[0044] Figure 5(b) shows a comparative schematic diagram of the power consumption of different models inference deployed using a certain chip according to an embodiment of the present disclosure.

[0045] Figure 6 This diagram illustrates a structural design of a low-bit-width and dynamic parameter initialization model training apparatus according to an embodiment of the present disclosure.

[0046] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0047] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0048] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0049] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0050] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0051] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0052] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0053] In related technologies, large language models suffer from high training costs, low training efficiency, limited deployment resources, and complex hardware scheduling. For example, traditional large model training methods require repeated trial and error to adjust hyperparameters, with a single training cycle costing up to $4.6 million. Furthermore, traditional growth-based training frameworks suffer from gradient propagation losses, requiring retraining of underlying parameters during model expansion, limiting knowledge inheritance efficiency, and resulting in lower training effectiveness. Traditional training paradigms rely on backpropagation optimization, with computational costs in the pre-training phase accounting for over 80% of the overall cost; even with mixed-precision computing, it remains difficult to overcome the bottleneck of balancing training efficiency and energy consumption. Full-precision models typically have high memory consumption and significant CPU inference latency, failing to meet the real-time requirements of edge devices. Finally, traditional low-bit quantization techniques still rely on high-precision gradient calculations during the training phase, failing to address the coordinated optimization of training energy consumption and inference efficiency, resulting in persistently high training costs. For example, traditional quantization methods such as Post-Training Quantization (PTQ) for 8-bit models have uncontrollable dynamic range of activation values, resulting in a performance drop of more than 15% after quantization. While mixed precision architectures alleviate the precision loss, they introduce hardware scheduling complexity and have limited improvement in inference speed on some resources.

[0054] To address one or more of the aforementioned technical problems, this disclosure provides a model training method that utilizes low bit width and dynamic parameter initialization. By integrating growth-based parameter expansion with low bit width quantization, the memory and GPU memory usage during training is significantly reduced, improving training efficiency, saving the cost of machine learning model training, and achieving efficient training of machine learning models. Simultaneously, the use of low bit width quantization for model parameters greatly reduces the memory and GPU memory occupied by the model, lowering the limitations on deployment resources. Furthermore, during the training phase, a first-precision computational kernel is used to handle the operations between the activation matrix and the weight matrix, and during the inference phase, a second-precision computational kernel is used to handle the operations between the activation matrix and the weight matrix, thereby achieving adaptive scheduling of computational kernels and reducing hardware scheduling complexity. The trained model can be deployed on different resources as needed; for example, resources with different precision computational kernels and small memory and GPU memory requirements can support model deployment.

[0055] For example, the application of low bit width and dynamic parameter initialization model training method provided in this disclosure can be executed by electronic devices such as terminal devices and servers, or by a part of an electronic device (such as a processor). The terminal device can be a desktop terminal or a mobile terminal, such as a laptop, tablet, desktop computer, smartphone, smart speaker, smartwatch, smart TV, in-vehicle terminal, or other types of electronic devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. As an example, the electronic device is configured with computing cores of different precision.

[0056] The following is a detailed description of a model training method using low bit width and dynamic parameter initialization provided in this disclosure.

[0057] Figure 1 A flowchart illustrating a method for training a model using low bit width and dynamic parameter initialization according to an embodiment of this disclosure is shown. Figure 1 As shown, the method may include the following steps:

[0058] Step 101: Obtain the parameter scale corresponding to the target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth.

[0059] The target machine learning model is the model to be trained, which can be any type of machine learning model without limitation; for example, the target machine learning model can be a large language model.

[0060] For example, the number of growth stages (also known as expansion stages) and the parameter size of each growth stage can be flexibly set according to the preset parameter size of the target machine learning model. The parameter size of each growth stage does not exceed the preset parameter size of the target machine learning model, and the parameter size of different growth stages gradually increases. For instance, if the preset parameter size of the target machine learning model is 10¹B (Billion), then three growth stages can be preset: the parameter size of the first growth stage can be 16B, the parameter size of the second growth stage can be 51B, and the parameter size of the third growth stage can be 10¹B. As another example, if the preset parameter size of the target machine learning model is 1T (Trillion), then five growth stages can be preset: the parameter size of the first growth stage is 16B, the parameter size of the second growth stage is 51B, the parameter size of the third growth stage is 100B, the parameter size of the fourth growth stage is 500B, and the parameter size of the fifth growth stage is 1T.

[0061] For each of the preset multiple growth stages, the parameter model of each growth stage can be expanded by performing the following steps, and the expanded model of each growth stage can be trained. In this way, by traversing the preset multiple growth stages, a trained target machine learning model can be obtained, thereby realizing the growth-based training of the target machine learning model through the preset multiple growth stages.

[0062] Step 102: Based on the parameter scale corresponding to the target growth stage, expand the parameter scale on the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the expanded parameter scale model are quantized using low bit width, and the low bit width means that the number of bits is less than the preset value of the bit width.

[0063] In this step, parameter scaling is performed on any one of the preset growth stages. Starting from the first of the preset growth stages, the parameter scale is expanded sequentially through each stage, thus completing a dynamic, phased parameter scaling process across the preset growth stages. After the parameter scaling in the last growth stage, the model's parameter scale reaches the preset parameter scale of the target machine learning model. Simultaneously, in each growth stage, parameter scaling is performed based on the model trained in the previous growth stage, thereby inheriting the knowledge learned in the previous stage, improving training efficiency, saving training costs, and ensuring the performance of the trained target machine learning model.

[0064] For example, models with expanded parameter scales at different growth stages have the same basic structure; the basic structure can be pre-defined as needed. Taking the target machine learning model as the target large language model as an example, the basic structure of existing large language models can be adopted, such as including: input layer, transformer architecture, output layer, etc.

[0065] For example, the number of network layers, hidden layer dimensions, and number of attention heads in the target large language model can be preset. In each growth stage, after expanding the parameter scale, the number of network layers, hidden layer dimensions, and number of attention heads in the model do not exceed the preset number of network layers, hidden layer dimensions, and number of attention heads. After the final growth stage, after expanding the parameter scale, the number of network layers, hidden layer dimensions, and number of attention heads in the model all reach the preset number of network layers, hidden layer dimensions, and number of attention heads. Here, the number of network layers can represent the number of Transformer Blocks, which, as the core unit of the Transformer architecture, are responsible for processing the contextual information of the input sequence. The hidden layer dimension represents the embedding vector dimension of each token in the large language model. The hidden layer dimension affects the model's information representation capability and model complexity. A higher hidden layer dimension allows the model to represent richer semantic information because a larger dimensional space can accommodate more feature details. The number of attention heads refers to the number of attention heads in the Transformer block. The Transformer block uses a multi-head self-attention mechanism. The number of attention heads is how many different "heads" the input sequence is divided into to perform self-attention calculations separately. Each attention head can learn different subspace features of the input sequence, allowing the model to capture information in the input sequence from multiple perspectives.

[0066] It should be noted that the first growth stage, as the initial growth stage, does not have a preceding growth stage. The trained model from the previous growth stage can be configured as empty. For example, in the first growth stage, the parameter scale is expanded based on a preset basic structure until it reaches the parameter scale corresponding to the first growth stage, thus completing the parameter scale expansion of the first growth stage. Alternatively, in the first growth stage, a preset model can be directly used as the model after parameter scale expansion. For example, an open-source model or a pre-designed model can be selected as the preset model, and the parameter scale of this preset model is the same as that corresponding to the first growth stage.

[0067] In this step, for any growth stage, after expanding the parameter scale in that growth stage, one or more weight parameters in the model can be quantized using a low bit width, so that all one or more weight parameters in the expanded parameter scale model are quantized using a low bit width. For example, each weight parameter newly added in this growth stage can be quantized using a low bit width. Here, low bit width means a bit width with fewer bits than a preset value, which can be 8 bits, 4 bits, 2 bits, etc.; for example, the low bit width can be 1.58 bits, thereby achieving ultra-low bit quantization.

[0068] In one possible implementation, the method further includes: initializing the newly added attention head and the weight parameters in the hidden layer dimension to any value in a preset set of values, wherein the preset set of values ​​includes three values: -1, 0, and 1.

[0069] For example, initializing the newly added attention head and the weight parameters in the hidden layer dimension to any value from a preset set of values ​​can include: in any growth stage, when expanding the parameter scale of the growth stage, for any weight parameter in the newly added attention head and the newly added hidden layer dimension during the width expansion, a value is randomly selected from the preset set of values ​​{-1, 0, +1} as the initial value of the weight parameter, and the value of the weight parameter occupies only 1.58 bits. In this way, for any growth stage, the weight ternary processing method is simultaneously applied to the newly added attention head and the hidden layer dimension, so that the newly added weight parameters in the model after expanding the parameter scale of the growth stage are quantized with a low bit width, thereby significantly saving the computing power and GPU memory usage required for subsequent model training in this growth stage.

[0070] In one possible implementation, expanding the parameter size based on the model trained in the previous growth stage of the target growth stage includes: expanding the parameter size of the model trained in the previous growth stage in two dimensions through width expansion and depth expansion; wherein, the width expansion is used to increase the number of attention heads and / or the hidden layer dimension in the model; and the depth expansion is used to increase the number of network layers in the model. Thus, in each growth stage, a two-dimensional expansion strategy using width expansion and depth expansion expands the model's parameters to the parameter size corresponding to that growth stage; by traversing multiple preset growth stages, the model size is gradually expanded to the preset parameter size of the target machine learning model.

[0071] For example, increasing the number of attention heads in the model when the width is expanded can include increasing the number of attention heads in each network layer (such as a Transformer block); wherein the number of attention heads in different network layers can be the same.

[0072] It should be noted that, for any growth stage, provided that the parameter size of the expanded model reaches the parameter size corresponding to that growth stage, both width expansion and depth expansion can be performed simultaneously to increase the number of attention heads, hidden layer dimensions, and network layers in the model. For example, in this growth stage, the number of attention heads, hidden layer dimensions, and network layers in the expanded model do not exceed the pre-set number of network layers, hidden layer dimensions, and attention heads in the target large language model.

[0073] For example, taking a target machine learning model with a preset parameter size of 101B and setting three growth stages: the first growth stage corresponds to a parameter size of 16B, the second growth stage to a parameter size of 51B, and the third growth stage to a parameter size of 101B. In the first growth stage, after expanding the parameter size, the model can have 64 network layers, 64 attention heads, and a hidden layer dimension of 8192. In the second growth stage, after expanding the parameter size, the model can have 80 network layers, 80 attention heads, and a hidden layer dimension of 10240. In the third growth stage, the model can have 140 network layers, 160 attention heads, and a hidden layer dimension of 20480.

[0074] In one possible implementation, the two-dimensional parameter scaling of the model trained in the previous growth stage through width scaling and depth scaling includes: during width scaling, randomly configuring the number of new attention heads and hidden layer dimensions in the model trained in the previous growth stage; and / or, during depth scaling, selecting one or more network layers in the model trained in the previous growth stage for replication based on the distance between the input and output sequences corresponding to each network layer in the model trained in the previous growth stage. Thus, in each growth stage, during width scaling, the number of new hidden dimensions and attention heads is randomly initialized, achieving dynamic parameter initialization; simultaneously, during depth scaling, the number of network layers in the model is increased by replicating existing network layers; thereby achieving two-dimensional parameter scaling with dynamic parameter initialization.

[0075] For example, when performing width expansion, randomly configuring the newly added attention heads and hidden layer dimensions in the model trained in the previous growth stage may include: randomly initializing the number of newly added attention heads and the number of newly added hidden dimensions without exceeding the preset number of attention heads and hidden dimensions.

[0076] For example, during depth expansion, based on the distance between the input and output sequences of each network layer in the model trained in the previous growth stage, one or more network layers in the model trained in the previous growth stage are selected for replication. This can include: for any growth stage, calculating the distance (such as Euclidean distance, cosine distance, etc.) between the input and output sequences of each network layer in the model trained in the previous growth stage, thereby obtaining the distances corresponding to each network layer, and then selecting the network layer with the smallest distance for replication, that is, selecting the network layer with the closest input and output sequence distance for replication; or, selecting multiple network layers for replication in ascending order of distance. The replicated network layers can be configured at the head or tail of the model, or adjacent to the original network layers, without limitation. In this way, by selecting the network layer with the smallest distance for replication based on the input-output distance of each network layer, depth expansion is completed, and the convergence of the model after expanding the parameter scale can be improved.

[0077] Step 103: Train the model after expanding the parameter scale, and call the first precision calculation kernel to process the operation between the activation matrix and the weight matrix during the training process to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, the second precision calculation kernel will be called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

[0078] In this step, after expanding the parameter model at any of the preset multiple growth stages, the model with expanded parameter scale is trained to obtain the trained model for that growth stage. This process is repeated across the preset multiple growth stages to obtain the trained target machine learning model, thus achieving growth-based training of the target machine learning model through multiple preset growth stages. It can be understood that for the first growth stage, if the preset model is directly used as the model with expanded parameter scale for the first growth stage, then training the model with expanded parameter scale in the first growth stage is equivalent to directly training the preset model.

[0079] In one possible implementation, at any growth stage, training data can be acquired to train the model after expanding the parameter scale for that growth stage. The training data can be unimodal data such as text or multimodal data such as text, images, and speech; appropriate training data can be selected based on requirements, without limitation. For example, training data can be selected from public data sources. Then, during the training of the model after expanding the parameter scale using the training data, matrix multiplication operations are performed on the weight matrix and activation matrix in the model based on the probability of text occurrences in the training data, thereby updating the weight values ​​in the weight matrix of the model. Then, after the training in this growth stage reaches a preset termination condition, the training task of this growth stage is completed, and the next growth stage begins, continuing parameter scale expansion and model training. The preset termination condition can include a preset number of iterations, a preset training duration, or model convergence, without limitation.

[0080] In one possible implementation, the method further includes optimizing the activation value distribution of the activation matrices corresponding to the copied one or more network layers through a preset transformation. For example, the preset transformation can be the Hadamard transformation, which is an orthogonal matrix with elements ±1. For example, optimizing the activation value distribution of the activation matrices corresponding to the copied one or more network layers through the preset transformation can include: during any growth stage, when training the model after expanding the parameter scale for that growth stage, optimizing the activation value distribution of the activation matrices corresponding to the network layers being depth-expanded and copied to a Gaussian distribution, or a Bayesian, Monte Carlo Tree Search (MCTS) distribution, etc., through the Hadamard transformation. For example, the activation matrix corresponding to the network layer can include the activation matrix output by the network layer. Thus, optimizing the activation value distribution of the activation matrix corresponding to the copied network layer through the Hadamard transformation reduces quantization error.

[0081] In one possible implementation, quantization-aware training (QAT) can be performed during the growth-oriented training process across multiple growth stages. QAT is a model optimization technique that simulates quantization errors during the training phase, aiming to allow the model to "adapt" to a low-precision inference environment in advance, thereby significantly reducing accuracy loss during deployment. For example, high-precision gradient calculation can be used during backpropagation when the model's parameter size is less than a preset threshold; and quantization-aware backpropagation can be initiated when the model's parameter size expands to exceed the preset threshold. Thus, during growth-oriented training, high-precision gradient calculation is used in the early stages to ensure the stability of model convergence, and quantization-aware backpropagation is gradually enabled as the model's parameter size expands. For example, taking a target machine learning model with a preset parameter size of 1T and five preset growth stages, where the parameter size for the first growth stage is 16B, the second growth stage is 51B, the third growth stage is 100B, the fourth growth stage is 500B, and the fifth growth stage is 1T, FP16 precision can be used to ensure model convergence stability during backpropagation in the early stages of growth training (such as the first growth stage). When the parameter size of the model expands to 100B (i.e., the third growth stage), quantization-aware backpropagation can be enabled.

[0082] For example, training the model with the expanded parameter size includes: during backpropagation, directly calculating the weights of the output layer weight matrix in the expanded parameter size model using the least squares method, and using a straight-through estimator (STE) to propagate the gradients of the hidden layer weight matrix in the expanded parameter size model. The hidden layer may include a network layer between the input and output layers of the model. Thus, for any growth stage using quantization-aware backpropagation, during backpropagation, the weight matrix of the output layer in the model after expanding the parameter scale of that growth stage is directly calculated using the least squares method, avoiding the accumulation of gradient errors caused by the traditional backpropagation method. For example, the feature matrix and target matrix of the output layer in the model can be directly calculated using the least squares solution to obtain the updated weights of the output layer. Since it is a one-time algebraic solution, there is no gradient backpropagation, thus avoiding backpropagation gradient errors. At the same time, for the weight matrix of the hidden layer in the model after expanding the parameter scale of that growth stage, the gradient is directly transmitted using a pass-through estimator, bypassing the quantization non-differentiable function. That is, for non-differentiable operations (such as round operation, argmax operation, etc.) in the quantization function, the non-differentiable operation is executed in the forward propagation, while the gradient is directly transmitted by skipping the non-differentiable operation in the backpropagation.

[0083] In one possible implementation, mixed-precision quantization calibration is performed on the model at each growth stage.

[0084] For example, the method further includes: storing each weight matrix in the model after expanding the parameter size and / or the model trained in the target growth stage using ternary quantization; and assigning quantization precision to each activation matrix according to the type of each activation matrix in the model after expanding the parameter size and / or the model trained in the target growth stage.

[0085] As an example, for any one of the preset multiple growth stages, after completing the expansion of the parameter scale for that growth stage, the weight matrices in the model after the expansion of the parameter scale for that growth stage are stored using ternary representation, and the quantization precision of each activation matrix in the model after the expansion of the parameter scale for that growth stage is assigned according to the type of each activation matrix in the model after the expansion of the parameter scale for that growth stage. As another example, for any one of the preset multiple growth stages, after completing the model training for that growth stage, the weight matrices in the trained model for that growth stage are stored using ternary representation, and the quantization precision of each activation matrix in the trained model for that growth stage is assigned according to the type of each activation matrix in the model after the training stage.

[0086] For example, when storing each weight matrix using ternary representation (-1, 0, 1), the values ​​of each weight in the weight matrix (i.e., -1, 0, or 1) can be stored in a lookup table for fast lookup. Each weight's value has a bit width of 1.58 bits. As an example, using ternary representation to store the weight matrices in a large language model trained through growth-based training at multiple growth stages significantly reduces the memory space occupied by the large language model, thus constructing a lightweight large language model.

[0087] For example, when allocating the quantization precision of each activation matrix, the quantization precision can be selected as needed; for example, the quantization precision can be FP16 (Half Precision Floating Point Number), and the quantized data is 16 bits wide; another example is INT8 (8-bit Integer), and the quantized data is 8 bits wide; yet another example is INT4 (4-bit Integer), and the quantized data is 4 bits wide.

[0088] For example, if an activation matrix is ​​an input to the attention layer (i.e., an activation matrix input to the attention layer), the quantization precision of the first bit width of the activation matrix can be assigned to quantize the activation values ​​in the matrix to the first bit width. If an activation matrix is ​​an intermediate state, such as an activation matrix output from the attention layer, the quantization precision of the second bit width can be assigned to quantize the activation values ​​in the matrix to the second bit width. The first bit width is less than the second bit width; for example, the first bit width can be 4 bits, and the second bit width can be 8 bits. As an example, 4-bit absolute maximum quantization (absmax) can be performed on the activation matrix input to the attention layer, while 8-bit sparsity can be used for the activation matrix in the intermediate state. This allows for dynamic allocation of quantization precision to the activation values ​​in the activation matrix based on its type.

[0089] For example, for any growth stage, during model training, the activation matrix maintains high precision, and a first-precision computation kernel is called to process the operations between the activation matrix and the weight matrix. Since quantization precision is assigned to each activation matrix in the trained model, when the model trained in this growth stage is subsequently used to perform inference tasks, a second-precision computation kernel will be called to process the operations between the activation matrix and the weight matrix. The first precision is greater than the second precision, thus achieving adaptive scheduling of the computation kernel and reducing hardware scheduling complexity. The trained model can be deployed on different resources as needed.

[0090] For example, when using the model trained in this growth stage to perform inference tasks, it can be configured with mixed-precision general matrix multiplication, which integrates a ternary lookup table (TL) kernel and a 4 / 8-bit activation dynamic scheduling module. Multiplication (mpGEMM) computational kernels, based on the quantization precision (e.g., 4-bit or 8-bit) allocated to the activation matrix, schedule computational kernels of corresponding precision to perform operations between the activation matrix and the weight matrix. This allows for model deployment with computational kernels of different precisions and relatively small memory and GPU memory requirements. For example, a trained model can be deployed on an electronic device with both 4-bit and 8-bit precision computational kernels to perform inference tasks. For an activation matrix with 8-bit quantization precision, the 4 / 8-bit activation dynamic scheduling module can schedule 8-bit precision computational kernels, and a ternary weight lookup table can be used to perform operations between the activation matrix and the ternary weight matrix. Similarly, for an activation matrix with 4-bit quantization precision, the 4 / 8-bit activation dynamic scheduling module can schedule 4-bit precision computational kernels, and a ternary weight lookup table can be used to perform operations between the activation matrix and the ternary weight matrix.

[0091] For example, based on the use of ternary storage for weights and dynamic allocation of precision for activation matrices according to type, compilers such as Triton and TVM (Tensor Virtual Machine) can be further utilized to retain the mask and quantization parameters through fusion functions. This ensures the compatibility of the output of the model after expanding the parameter scale at each growth stage and / or the output of the trained model with the quantized version, that is, it ensures that the quantization parameters corresponding to the same activation matrix before and after expanding the parameter scale remain consistent.

[0092] In one possible implementation, outlier channels are reordered for each growth stage.

[0093] For example, the method further includes: detecting outlier channels in the activation matrix; arranging each detected outlier channel at the end of the activation matrix containing the outlier channel to obtain a reordered activation matrix corresponding to the outlier channel; and synchronously adjusting the positions of each channel in the associated weight matrix based on the positions of each channel in the reordered activation matrix corresponding to the outlier channel, so that the adjusted weight matrix matches the reordered activation matrix corresponding to the outlier channel; and assigning different quantization precipitates to the outlier channel and the normal value channels in the activation matrix containing the outlier channel. Thus, by employing outlier channel reordering technology, through dynamic reordering of outlier channels in the activation matrix, combined with a static reordering strategy for the weight matrix, and by assigning separate quantization precipitates to outlier channels, quantization errors are effectively reduced, model accuracy is improved, and the inherent contradiction between model accuracy and resource compression is overcome.

[0094] The outlier channel in the activation matrix refers to the channel containing the outlier activation value that is significantly larger than other activation values.

[0095] For example, outlier channel detection of activation matrices may include: detecting outlier channels of each activation matrix in the model during the expansion of parameter size at each growth stage, or detecting outlier channels of each activation matrix in the model during the training of the expanded parameter size model at each growth stage, or detecting outlier channels of each activation matrix in the model after expanding parameter size at each growth stage, or detecting outlier channels of each activation matrix in the model after training at each growth stage.

[0096] For example, each detected outlier channel is arranged at the end of the activation matrix containing the outlier channel to obtain the reordered activation matrix corresponding to the outlier channel. Based on the position of each channel in the reordered activation matrix corresponding to the outlier channel, the position of each channel in the associated weight matrix is ​​simultaneously adjusted so that the adjusted weight matrix is ​​adapted to the reordered activation matrix corresponding to the outlier channel. This can include: arranging each detected outlier channel at the end of the activation matrix containing the outlier channel to dynamically reorder the outlier channels; and simultaneously, for the weight matrix (i.e., the associated weight matrix) that needs to be multiplied with the activation matrix containing the outlier channel, static reordering of each channel is performed simultaneously to adapt it to each channel in the reordered activation matrix. In this way, the correspondence between each channel when performing matrix multiplication between the reordered activation matrix and the reordered associated weight matrix is ​​consistent with the correspondence between each channel when performing matrix multiplication between the activation matrix before reordering and the associated weight matrix before reordering, thereby ensuring the computational equivalence of matrix multiplication.

[0097] For example, assigning different quantization precisions to the outlier channel and the normal value channel in the activation matrix where the outlier channel is located can include assigning a higher quantization precision to the outlier channel than to the normal value channel; thus, considering the impact of outliers on model performance, assigning higher quantization precision to the outlier channel in the activation matrix can effectively reduce quantization error.

[0098] As an example, for the activation matrix of the input attention layer, if outlier channels are detected, the quantization precision of the first bit width of the outlier channel is allocated to quantize the activation values ​​in the outlier channel to the first bit width. Simultaneously, the quantization precision of the second bit width of the other channels (i.e., normal value channels) in the activation matrix is ​​allocated to the second bit width, where the first bit width is greater than the second bit width; for example, the first bit width can be 4 bits and the second bit width can be 8 bits. Thus, during growth training, this outlier channel reordering technique dynamically reorders the outlier channels in the activation matrix, concentrating them at the end of the activation matrix and allocating 8 bits of quantization precision to each outlier channel, while assigning 4 bits of quantization precision to the other normal value channels. The associated weight matrices are also statically reordered synchronously to ensure the computational equivalence of matrix multiplication. Compared to using 4 bits of quantization for all channels in the activation matrix (including outlier and normal value channels), this reduces quantization error by 42%. Meanwhile, after evaluation using the mathematical reasoning task (GSM8K, Grade School Math 8K), the use of this anomalous channel reordering technique still maintained an accuracy of 58.38%, exceeding the full-precision model of the same scale by 1.2 percentage points.

[0099] In one possible implementation, in addition to reordering abnormal channels, a residual approximation strategy can be further adopted. The residual approximation strategy uses the output of the previous growth stage as the teacher signal. Through the weighted fusion of language modeling loss and quantization error loss, the performance difference of the model after fine-tuning after expanding the parameter scale is less than 0.1%, which is significantly better than the 15% accuracy loss of traditional post-training quantization methods.

[0100] In this embodiment of the disclosure, the parameter scale corresponding to the target growth stage is obtained through steps 101-103 above; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth; based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the expanded parameter scale model are quantized using a low bit width, the low bit width indicating a bit width with a number of bits less than a preset value; the expanded parameter scale model is trained, and during the training process, a first precision calculation kernel is called to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform an inference task, a second precision calculation kernel is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision. In this way, by combining growth-based parameter expansion with low-bit-width quantization, the memory and GPU memory usage during training is significantly reduced, training efficiency is improved, and the cost of training machine learning models is saved, achieving efficient training of machine learning models. Simultaneously, the use of low-bit-width quantization for model parameters greatly reduces the memory and GPU memory occupied by the model, lowering the limitations on deployment resources. Furthermore, the use of a first-precision computational kernel to handle the operations between the activation and weight matrices during the training phase and a second-precision computational kernel during the inference phase enables adaptive scheduling of computational kernels, reducing hardware scheduling complexity. The trained model can be deployed on different resources as needed; for example, resources with different precision computational kernels and low memory and GPU memory requirements can support model deployment.

[0101] Among them, growth-type parameter expansion, also known as dynamic parameter expansion or low-rank parameter expansion, is a method for training large models that reduces computational requirements. It reduces the computational and GPU memory usage of large model training, thereby reducing the hardware required and training costs. Verification has shown that using the method described in this embodiment to train a model with hundreds of billions of parameters reduces the total cost to the $100,000 level, a reduction of over 90% compared to traditional methods, with a simultaneous 95% reduction in GPU memory usage (for example, a 2B parameter model requires only 0.4GB).

[0102] Figure 2 A schematic diagram illustrating the training loss of a growth training model according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the model is divided into three growth stages: growth stage 1, growth stage 2, and growth stage 3. Growth stage 1 corresponds to a parameter scale of 16 bytes; growth stage 2 corresponds to a parameter scale of 51 bytes; and growth stage 3 corresponds to a parameter scale of 101 bytes. Using the training method described in this embodiment, a model with a parameter scale of 101 bytes is gradually trained from a 16-byte parameter scale through growth stages 1, 2, and 3. Figure 2 The data shows the trend of training loss as a function of the number of processed tokens. It can be seen that the training loss decreases rapidly during training, and the ideal training effect can be achieved with a smaller number of tokens, which improves training efficiency and saves training costs.

[0103] Figure 3(a) shows a schematic diagram of the model training loss variation curve under different hyperparameter combinations according to an embodiment of the present disclosure. As shown in Figure 3(a), different hyperparameter combinations are searched using the grid search method, and training is performed based on different hyperparameter combinations using the method described in the embodiment of the present disclosure. The figure shows the trend of the model training loss as a function of training steps. Different colors represent different hyperparameter combinations. It can be seen that when training based on different hyperparameter combinations using the method described in the embodiment of the present disclosure, the training loss continues to decrease, and the loss function can converge quickly, achieving the ideal learning effect.

[0104] Figure 3(b) shows a schematic diagram of the gradient vector norm variation curves under different hyperparameter combinations according to an embodiment of the present disclosure. As shown in Figure 3(b): different hyperparameter combinations are searched using a grid search method, and the above-mentioned method in the embodiment of the present disclosure is used for training based on different hyperparameter combinations. The gradient vector norm can be seen to change with training deployment, where different colors represent different hyperparameter combinations. It can be seen that by using the above-mentioned method in the embodiment of the present disclosure for training based on different hyperparameter combinations, satisfactory results can be achieved in terms of gradient "magnitude" and "stability" after a relatively short number of training steps.

[0105] Figure 4 This diagram illustrates the effect of low-bit quantization according to an embodiment of the present disclosure, such as... Figure 4 As shown, for large language models with the same parameter size, when training the large language model to the same performance, the training cost of the low-bit quantization large language model obtained using the training method in this embodiment is significantly lower than the training cost of the traditional large language model using FP16 precision. Furthermore, because this embodiment uses low-bit quantization, the difficulty of multiplication operations between the weight matrix and activation matrix during inference is reduced, thereby lowering the resource requirements for model deployment. Traditional FP16 precision large language models typically require deployment on GPUs, while the large language model obtained using the training method in this embodiment can also be deployed on CPUs with x86 / ARM architectures.

[0106] For example, for a large language model with a preset parameter size of 1T, the method described above in this embodiment is used for training. Through width-depth dual-dimensional expansion at each growth stage, the model gradually grows from a 16B parameter size to a 1T parameter size. During width expansion, the newly added hidden layer dimensions and attention heads are randomly initialized, and weight trivaluation preprocessing is applied simultaneously. During depth expansion, the layer with the closest input-output distance is selected for replication, and the replicated parameters are optimized for activation distribution using Hadamard transformation to reduce quantization error. Thus, the model starts with a 16B parameter size and gradually grows to a 1T size through width expansion (random initialization of newly added hidden layer dimensions) and depth expansion (layer replication based on input-output distance). Each growth stage embeds weight trivaluation preprocessing ({-1, 0, +1}, 1.58 bits / parameter), combined with Hadamard transformation to optimize activation distribution, reducing quantization error by 42%. Furthermore, after each growth stage is completed, the model is calibrated using mixed-precision quantization: weights are stored in ternary form (1.58 bits), and activation values ​​are dynamically assigned precision according to the layer type (attention layer inputs use 4-bit absmax quantization, and intermediate states use 8-bit sparsity); the mask and quantization parameters are preserved through the fusion function to ensure compatibility between the expanded model output and the quantized version.

[0107] Furthermore, after completing growth-based training with multiple preset growth stages, the weights and quantization parameters of the trained target machine learning model can be encapsulated in a preset format. For example, the GGUF (GPT-Generated Unified Format) format can be used to uniformly encapsulate the weights and quantization parameters of the trained target machine learning model. GGUF is a binary file format designed for large language models, and its core value lies in unifying the original model weights and quantization parameters into a single file, achieving efficient deployment and cross-platform compatibility. Consequently, the encapsulated model can be deployed on different platforms; for example, it can support deployment across all platforms from native 4-bit computing platforms to Apple's M-series chips, improving the model's hardware ecosystem compatibility. Verification has shown that in long text processing scenarios, using low-bit quantization combined with post-quantization techniques such as RoPE reduces the memory footprint of the 16K context by 80%, and stabilizes the inference latency on edge devices within 120ms. This integrated training-inference design overcomes the contradiction between the expansion rules and the growth rate of hardware resources in traditional frameworks, providing a feasible path for the deployment of trillion-parameter models on consumer devices.

[0108] Furthermore, this disclosure also provides a mixed-precision general matrix multiplication (mpGEMM) computation core. A unified mixed-precision computation core (mpGEMM) can be developed using existing tools, integrating a ternary lookup table (TL) core and a 4 / 8-bit activation dynamic scheduling module. During growth-based training through multiple growth stages, the FP16 core is preferentially used to expand the model parameter scale, while during the inference stage, it can automatically switch to a low-level quantization core; thus achieving hardware adaptive scheduling and collaborative training and inference. Specifically, during the inference stage, the mixed-precision computation core achieves a 6.46x speedup on x86 / ARM architectures, with a single CPU reaching a human-readable response speed of 5-7 tokens / second. Under certain parameter scales, when deployed on chips such as the Apple M2 for inference, the power consumption is as low as 0.028 joules / token.

[0109] Figure 5(a) illustrates a comparative diagram of the inference speed of different models deployed on a certain chip according to an embodiment of the present disclosure. As shown in Figure 5(a), the inference speed of a low-bit quantization model trained using the method described in the embodiment of the present disclosure and deployed on an Apple M2 chip is compared with that of a control model (such as llama.cpp) using conventional quantization with FP16 or INT4 and deployed on an Apple M2 chip. Using a human reading speed of 5-7 tokens / s as a benchmark, and with the same model parameter size, the inference speed of the low-bit quantization model trained using the method described in the embodiment of the present disclosure is significantly faster than that of the control model. Furthermore, when the model parameter size reaches 100B, the inference speed of the control model is significantly lower than the human reading speed, while the inference speed of the low-bit quantization model trained using the method described in the embodiment of the present disclosure is still comparable to the human reading speed.

[0110] Figure 5(b) illustrates a comparative diagram of the inference power consumption of different models deployed on a certain chip according to an embodiment of the present disclosure. As shown in Figure 5(b), the inference power consumption of a low-bit quantization model trained using the method described in the embodiment of the present disclosure on an Apple M2 chip is compared with that of a control model (such as llama.cpp) using conventional FP16 or INT4 quantization deployed on an Apple M2 chip. When the parameter scale of the models is 700M, the power consumption of the low-bit quantization model trained using the method described in the embodiment of the present disclosure is reduced by 55.4% compared to the control model. When the parameter scale of the models is 70B, the power consumption of the low-bit quantization model trained using the method described in the embodiment of the present disclosure is reduced by 70% compared to the control model. It is evident that the model trained using the method described in the embodiment of the present disclosure can significantly save power.

[0111] Thus, in some scenarios, during training by combining growth-based parameter expansion with low-bit-width quantization, dynamic parameter initialization, quantization-aware training, and mixed precision can be combined to achieve a breakthrough balance between training efficiency, resource compression, and model performance, resulting in significant computational savings for large-scale model training and inference. This addresses the following contradictions faced by existing lightweight large language models: the positive correlation between training cost and model size (existing scaling laws require parameter size to grow in tandem with data volume, but the growth rate of hardware resources lags behind the demand for increased model complexity); the antagonism between maintaining model accuracy and resource compression (quantization techniques generally face the impossible triangle of "compression rate-accuracy-energy consumption," requiring structural innovation to achieve bit-level computational reconstruction); and the reliance on static masks in training function retention mechanisms, which are difficult to adapt to dynamic topology adjustments with trillions of parameters.

[0112] Based on the same inventive concept of the above method embodiments, the present disclosure also provides a model training device that applies low bit width and dynamic parameters to initialize the model training. This device can be used to execute the technical solutions described in the above method embodiments.

[0113] Figure 6 This diagram illustrates a structural design of a model training apparatus for low bit width and dynamic parameter initialization according to an embodiment of the present disclosure. Figure 6 As shown, the device includes: an acquisition module 601, used to acquire the parameter scale corresponding to a target growth stage; wherein the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth; an expansion module 602, used to expand the parameter scale based on the parameter scale corresponding to the target growth stage and on the model trained in the previous growth stage of the target growth stage; wherein one or more parameters in the expanded parameter scale model are quantized using a low bit width, the low bit width indicating a bit width with a number of bits less than a preset value; a training module 603, used to train the model with the expanded parameter scale, and during the training process, calling a first precision calculation kernel to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein when the model trained in the target growth stage is used to perform an inference task, a second precision calculation kernel will be called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

[0114] In this embodiment, the parameter scale corresponding to the target growth stage is obtained. The target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model. The plurality of growth stages are used to train the target machine learning model through growth-based training. Based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage. In the expanded parameter scale model, one or more parameters are quantized using low bit width, where low bit width means a bit width with fewer bits than a preset value. The expanded parameter scale model is trained, and during the training process, a first-precision computation kernel is called to process the operation between the activation matrix and the weight matrix, resulting in the model trained in the target growth stage. When the model trained in the target growth stage is used to perform inference tasks, a second-precision computation kernel is called to process the operation between the activation matrix and the weight matrix. The first precision is greater than the second precision. Thus, by integrating growth-based parameter expansion and low-bit-width quantization, the memory and GPU memory usage during training is significantly reduced, training efficiency is improved, the cost of training the machine learning model is saved, and efficient training of the machine learning model is achieved. Meanwhile, the model uses low-bit-width quantization for parameters, which greatly reduces the memory and GPU memory occupied by the model and lowers the limitations on deployment resources. Furthermore, during the training phase, a first-precision computing kernel is used to handle the operations between the activation matrix and the weight matrix, and during the inference phase, a second-precision computing kernel is used to handle the operations between the activation matrix and the weight matrix, thereby achieving adaptive scheduling of computing kernels and reducing hardware scheduling complexity. The trained model can be deployed on different resources as needed. For example, resources with different precision computing kernels and small memory and GPU memory can support the deployment of the model.

[0115] In one possible implementation, the expansion module 602 is further configured to: expand the model trained in the previous growth stage in two dimensions through width expansion and depth expansion; wherein, the width expansion is used to increase the number of attention heads and / or the hidden layer dimension in the model; and the depth expansion is used to increase the number of network layers in the model.

[0116] In one possible implementation, the expansion module 602 is further configured to: randomly configure the number of attention heads and hidden layer dimensions added to the model trained in the previous growth stage when performing the width expansion; and / or, when performing the depth expansion, select one or more network layers in the model trained in the previous growth stage for replication based on the distance between the input sequence and the output sequence corresponding to each network layer in the model trained in the previous growth stage.

[0117] In one possible implementation, the extension module 602 is further configured to: initialize the newly added attention head and the weight parameters in the hidden layer dimension to any value in a preset set of values, wherein the preset set of values ​​includes three values: -1, 0, and 1; and / or optimize the activation value distribution of the activation matrix corresponding to the one or more network layers being copied through a preset transformation.

[0118] In one possible implementation, the training module 603 is further configured to: store each weight matrix in the model after the parameter scaling is extended and / or the model trained in the target growth stage using ternary quantization; and allocate the quantization precision of each activation matrix according to the type of each activation matrix in the model after the parameter scaling is extended and / or the model trained in the target growth stage.

[0119] In one possible implementation, the training module 603 is further configured to: detect outlier channels in the activation matrix; arrange each detected outlier channel at the end of the activation matrix where the outlier channel is located to obtain the reordered activation matrix corresponding to the outlier channel; and based on the position of each channel in the reordered activation matrix corresponding to the outlier channel, synchronously adjust the position of each channel in the associated weight matrix so that the adjusted weight matrix is ​​adapted to the reordered activation matrix corresponding to the outlier channel; and assign different quantization precisions to the outlier channel and the normal value channels in the activation matrix where the outlier channel is located.

[0120] In one possible implementation, the training module 603 is further configured to: during backpropagation, directly calculate the weights of the output layer weight matrix in the model after the parameter scaling is extended using the least squares method, and pass the gradients of the hidden layer weight matrix in the model after the parameter scaling is extended using a pass-through estimator.

[0121] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0122] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0123] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0124] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0125] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 7 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0126] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0127] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0130] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0131] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for initializing a model training process using low bit width and dynamic parameters, characterized in that, The method includes: Obtain the parameter scale corresponding to the target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth. Based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the expanded parameter scale model are quantized with low bit width, and the low bit width means that the number of bits is less than the preset value of the bit width. The model with the expanded parameter size is trained, and during the training process, a first-precision computation kernel is called to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision computation kernel is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

2. The method according to claim 1, characterized in that, Expanding the parameter scale based on the model trained in the previous growth stage of the target growth stage includes: The model trained in the previous growth stage is expanded in two dimensions by width expansion and depth expansion; wherein, the width expansion is used to increase the number of attention heads and / or the hidden layer dimension in the model; and the depth expansion is used to increase the number of network layers in the model.

3. The method according to claim 2, characterized in that, The process of expanding the parameter scale of the model trained in the previous growth stage through width and depth expansion includes: When performing the width expansion, the number of new attention heads and the hidden layer dimension are randomly configured in the model trained in the previous growth stage. And / or, During the depth expansion, based on the distance between the input and output sequences of each network layer in the model trained in the previous growth stage, one or more network layers in the model trained in the previous growth stage are selected for replication.

4. The method according to claim 3, characterized in that, The method further includes: The newly added attention head and the weight parameters in the hidden layer dimension are initialized to any value in a preset set of values, wherein the preset set of values ​​includes three values: -1, 0, and 1. And / or, The activation value distribution of the activation matrix corresponding to the one or more network layers is optimized through a preset transformation.

5. The method according to claim 4, characterized in that, The method further includes: The weight matrices in the model after the parameter scaling is ternary and / or in the model trained during the target growth stage are stored using ternary representation. The quantization precision of each activation matrix is ​​assigned based on the type of each activation matrix in the model after the expansion parameter scale and / or in the model trained in the target growth stage.

6. The method according to claim 4 or 5, characterized in that, The method further includes: Perform outlier channel detection on the activation matrix; Each detected outlier channel is placed at the end of the activation matrix where the outlier channel is located, resulting in the reordered activation matrix corresponding to the outlier channel. Based on the position of each channel in the reordered activation matrix corresponding to the outlier channel, the position of each channel in the associated weight matrix is ​​adjusted synchronously so that the adjusted weight matrix is ​​adapted to the reordered activation matrix corresponding to the outlier channel. Assign different quantization precisions to the outlier channel and the normal value channel in the activation matrix containing the outlier channel.

7. The method according to claim 1, characterized in that, Training the model after expanding the parameter scale includes: During backpropagation, the weights of the output layer in the model with the expanded parameter size are directly calculated using the least squares method, and the gradients of the hidden layer in the model with the expanded parameter size are passed through an estimator.

8. A model training device that uses low bit width and dynamic parameter initialization, characterized in that, The device includes: An acquisition module is used to acquire the parameter scale corresponding to a target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth. The extension module is used to expand the parameter scale based on the parameter scale corresponding to the target growth stage and the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the model after expanding the parameter scale are quantized with low bit width, and the low bit width means that the number of bits is less than a preset value. The training module is used to train the model after expanding the parameter scale, and during the training process, it calls a first-precision calculation kernel to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision calculation kernel will be called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images