Model training method applying low bit width and dynamic parameter initialization

Abstract

The invention relates to the technical field of artificial intelligence, and provides a method for initializing model training by applying low bit width and dynamic parameters, comprising the following steps: acquiring a parameter scale corresponding to a target growth stage; based on the parameter scale corresponding to the target growth stage, expanding the parameter scale on the basis of the model trained in the previous growth stage of the target growth stage; quantizing one or more parameters in the model after the parameter scale is expanded by adopting a low bit width; training the model after the parameter scale is expanded, and calling a first precision calculation core to process the operation between the activation matrix and the weight matrix in the training process to obtain a trained model in the target growth stage; wherein when the model trained in the target growth stage is used for executing the reasoning task, the second precision calculation core is called to process the operation between the activation matrix and the weight matrix. Through the method and the device, the cost of model training is saved, the limitation on deployment resources is reduced, and the hardware scheduling complexity is reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a model training method using low bit width and dynamic parameter initialization. BACKGROUND

[0002] For large language models, the existing training methods generally have high training costs. Moreover, the parameter scale of a large language model is usually large, and the memory or display memory is occupied more, so that the deployment resources are limited; and although some model quantization can reduce the memory or display memory occupation to a certain extent, it increases the hardware scheduling complexity. SUMMARY

[0003] Therefore, the present disclosure provides a model training method using low bit width and dynamic parameter initialization, a device, an electronic device, a storage medium and a computer program product.

[0004] According to an aspect of the present disclosure, a model training method using low bit width and dynamic parameter initialization is provided, and the method comprises:

[0005] obtaining a parameter scale corresponding to a target growth stage; wherein the target growth stage is any growth stage in a plurality of preset growth stages, and the parameter scale corresponding to each growth stage in the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; and the plurality of growth stages are used to train the target machine learning model by growth training;

[0006] extending the parameter scale based on the model trained in the previous growth stage of the target growth stage; wherein one or more parameters in the model after extending the parameter scale are quantized using low bit width, and the low bit width represents a bit width with a bit number less than a preset value;

[0007] training the model after extending the parameter scale, and calling a first precision calculation kernel to process the operation between an activation matrix and a weight matrix in a training process to obtain a model trained in the target growth stage; wherein when the model trained in the target growth stage is used to execute an inference task, a second precision calculation kernel is called to process the operation between the activation matrix and the weight matrix; and the first precision is greater than the second precision.

[0008] In a possible implementation manner, the extending the parameter scale based on the model trained in the previous growth stage of the target growth stage comprises:

[0009] The model trained in the previous growth stage is expanded in two dimensions in parameter scale through width expansion and depth expansion; the width expansion is used to increase the number of attention heads and / or the dimension of the hidden layer in the model; and the depth expansion is used to increase the number of network layers in the model.

[0010] In a possible implementation, the expanding the model trained in the previous growth stage in two dimensions in parameter scale through width expansion and depth expansion comprises:

[0011] When the width expansion is performed, the number of newly added attention heads and the dimension of the hidden layer in the model trained in the previous growth stage are randomly configured;

[0012] and / or,

[0013] When the depth expansion is performed, one or more network layers in the model trained in the previous growth stage are selected to be copied based on the distance between the input sequence and the output sequence corresponding to each network layer in the model trained in the previous growth stage.

[0014] In a possible implementation, the method further comprises:

[0015] The weight parameters in the newly added attention heads and the hidden layer are initialized as any value in a preset value set, wherein the preset value set includes three values: -1, 0 and 1.

[0016] and / or,

[0017] The activation value distribution of the activation matrix corresponding to the one or more copied network layers is optimized through a preset transformation.

[0018] In a possible implementation, the method further comprises:

[0019] The model after the expansion in parameter scale and / or each weight matrix in the model trained in the target growth stage is stored in ternarization;

[0020] The quantization precision of each activation matrix is allocated according to the type of each activation matrix in the model after the expansion in parameter scale and / or the model trained in the target growth stage.

[0021] In a possible implementation, the method further comprises:

[0022] An outlier channel detection of the activation matrix is performed;

[0023] arranging each detected outlier channel at the end of the activation matrix in which the outlier channel is located to obtain a reordered activation matrix corresponding to the outlier channel; and synchronously adjusting the positions of the channels in the associated weight matrix based on the positions of the channels in the reordered activation matrix corresponding to the outlier channel, so that the adjusted weight matrix is adapted to the reordered activation matrix corresponding to the outlier channel.

[0024] allocating different quantization precisions for the outlier channel and the normal value channels of the activation matrix in which the outlier channel is located.

[0025] In a possible implementation, the training of the model with the expanded parameter scale comprises:

[0026] In the process of back propagation, the weight matrix of the output layer in the model with the expanded parameter scale is directly calculated by using the least square method, and the weight matrix of the hidden layer in the model with the expanded parameter scale is transmitted by using the straight-through estimator.

[0027] According to another aspect of the present disclosure, there is provided a device for training a model with low bit width and dynamic parameter initialization, which comprises:

[0028] The acquisition module is configured to acquire a parameter scale corresponding to a target growth stage, wherein the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each growth stage of the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; the plurality of growth stages are used to train the target machine learning model by growth training.

[0029] The expansion module is configured to expand the parameter scale based on the parameter scale corresponding to the target growth stage on the basis of a model trained in a previous growth stage of the target growth stage, wherein one or more parameters in the model with the expanded parameter scale are quantized by using a low bit width, and the low bit width represents a bit width with a bit number less than a preset value.

[0030] The training module is configured to train the model with the expanded parameter scale and to call a first precision calculation core to process the operation between the activation matrix and the weight matrix during the training to obtain a model trained in the target growth stage; wherein when the model trained in the target growth stage is used to perform an inference task, a second precision calculation core is called to process the operation between the activation matrix and the weight matrix; and the first precision is greater than the second precision.

[0031] According to another aspect of the present disclosure, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0032] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.

[0033] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-transitory computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0034] According to aspects of the present disclosure, a parameter scale corresponding to a target growth stage is obtained; wherein the target growth stage is any one of a plurality of preset growth stages, and a parameter scale corresponding to each growth stage of the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth training; based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded based on a model trained in a previous growth stage of the target growth stage; wherein one or more parameters in the model after the parameter scale is expanded are quantized using a low bit width, and the low bit width represents a bit width with a bit number less than a preset value.

[0035] The model after the parameter scale is expanded is trained, and a first precision calculation kernel is called to process operations between an activation matrix and a weight matrix during the training process to obtain a model trained in the target growth stage; wherein when the model trained in the target growth stage is used to perform an inference task, a second precision calculation kernel is called to process operations between the activation matrix and the weight matrix; and the first precision is greater than the second precision. In this way, by fusing growth training and low bit width quantization, the memory and video memory occupancy during the training process is significantly reduced, the training efficiency is improved, the cost of machine learning model training is saved, and efficient training of the machine learning model is achieved. At the same time, the parameters in the model are quantized using a low bit width, which greatly reduces the memory and video memory occupied by the model and reduces the limitation on deployment resources. Furthermore, the first precision calculation kernel is called to process operations between the activation matrix and the weight matrix during the training phase, and the second precision calculation kernel is called to process operations between the activation matrix and the weight matrix during the inference phase, thereby achieving adaptive scheduling of the calculation kernel and reducing the hardware scheduling complexity. The trained model can be deployed on different resources as needed, for example, resources with different precision calculation kernels and smaller memory and video memory can support the deployment of the model.

[0036] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present disclosure, and together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 FIG. 1 shows a flowchart of a method for training a model using a low bit-width and dynamic parameter initialization according to an embodiment of the present disclosure.

[0039] Figure 2 FIG. 2 shows a diagram of training loss of a growing model according to an embodiment of the present disclosure.

[0040] FIG. 3(a) shows a diagram of model training loss variation curves under different hyperparameter combinations according to an embodiment of the present disclosure.

[0041] FIG. 3(b) shows a diagram of gradient vector norm variation curves under different hyperparameter combinations according to an embodiment of the present disclosure.

[0042] Figure 4 FIG. 4 shows a diagram of the effect of low bit-width quantization according to an embodiment of the present disclosure.

[0043] FIG. 5(a) shows a diagram of comparison of inference speeds of different models deployed on a certain chip according to an embodiment of the present disclosure.

[0044] FIG. 5(b) shows a diagram of comparison of inference energy consumptions of different models deployed on a certain chip according to an embodiment of the present disclosure.

[0045] Figure 6 FIG. 6 shows a block diagram of a device for training a model using a low bit-width and dynamic parameter initialization according to an embodiment of the present disclosure.

[0046] Figure 7 FIG. 7 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Various example embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements or components having a substantially the same function or operation. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0048] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", "provide", "providing", or variants thereof, are open-ended, and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0049] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements can be present.

[0050] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0051] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0052] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known devices, methods, procedures, components, and circuits will not be described in detail herein. It will be apparent to those skilled in the art that the present inventive concept can be practiced without these specific details.

[0053] In the related art, for large language models, there are problems such as high training cost, low training efficiency, limited deployment resources, and complex hardware scheduling: for example, the traditional large model training method needs to adjust the hyperparameters through repeated trial and error, and the single training cost is as high as 4.6 million US dollars. For another example, the traditional growing training framework has a gradient transmission loss problem, which causes the model to need to retrain the bottom layer parameters when expanding, and the knowledge inheritance efficiency is limited, and the training effect is low. For another example, the traditional training paradigm relies on back propagation optimization, and the computing power consumption in the pre-training stage accounts for more than 80% of the overall cost. Even if mixed precision calculation is used, it is still difficult to break through the balance bottleneck of training efficiency and energy consumption. For another example, the memory of the full-precision model usually occupies a high proportion, and the CPU inference delay is large, which cannot meet the real-time demand of edge devices. For another example, the traditional low-bit quantization technology still relies on high-precision gradient calculation in the training stage, and does not solve the problem of cooperative optimization of training energy consumption and inference efficiency, and the training cost is still high. For another example, the traditional 8-bit model post-training quantization (PTQ) and other quantization methods cause the performance of the quantized model to decrease by more than 15% due to the uncontrollable dynamic range of activation values, and although the mixed precision architecture alleviates the precision loss, it introduces hardware scheduling complexity, and the inference speed on some resources is limited.

[0054] To solve one or more of the above technical problems, the present disclosure provides a low-bit-width and dynamic parameter initialization model training method. By fusing the growing parameter expansion and low-bit-width quantization, the memory and video memory occupation in the training process is significantly reduced, the training efficiency is improved, the cost of machine learning model training is saved, and efficient training of the machine learning model is realized. At the same time, the parameters in the model are quantized using low-bit-width, which greatly reduces the memory and video memory occupied by the model, reduces the limitation on deployment resources, and in the training stage, a first-precision calculation core is called to process the operation between the activation matrix and the weight matrix, and in the inference stage, a second-precision calculation core is called to process the operation between the activation matrix and the weight matrix, thereby realizing adaptive scheduling of the calculation core and reducing the hardware scheduling complexity; the trained model can be deployed on different resources as needed, for example, a resource with a calculation core of different precision and smaller memory and video memory can support the deployment of the model.

[0055] Exemplarily, the low-bit-width and dynamic parameter initialization model training method provided by the present disclosure can be executed by a terminal device, a server, or a part (such as a processor) of an electronic device, wherein the terminal device can be a desktop terminal or a mobile terminal, for example, various types of electronic devices such as a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart speaker, a smart watch, a smart television, and a vehicle-mounted terminal. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms, etc. Basic cloud computing services such as platform. As an example, the electronic device is configured with calculation cores of different precisions.

[0056] A low-bit-width and dynamic parameter initialization model training method provided by the present disclosure is described in detail below.

[0057] Figure 1 A flowchart of a low-bit-width and dynamic parameter initialization model training method according to an embodiment of the present disclosure is shown. As shown in Figure 1 The method can include the following steps:

[0058] Step 101, obtaining a parameter scale corresponding to a target growth stage; wherein the target growth stage is any growth stage in a plurality of preset growth stages, and the parameter scale corresponding to each growth stage in the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; the plurality of growth stages are used to train the target machine learning model by growing.

[0059] The target machine learning model is a model to be trained, which can be various types of machine learning models, and is not limited. For example, the target machine learning model can be a large language model.

[0060] For example, the number of growth stages (also referred to as expansion stages) and the parameter scale corresponding to each growth stage can be flexibly set according to the preset parameter scale of the target machine learning model. The parameter scale corresponding to each growth stage is not more than the preset parameter scale of the target machine learning model, and the parameter scales corresponding to different growth stages gradually increase. For example, if the preset parameter scale of the target machine learning model is 101B (Billion), three growth stages can be preset, the parameter scale corresponding to the first growth stage can be 16B, the parameter scale corresponding to the second growth stage can be 51B, and the parameter scale corresponding to the third growth stage can be 101B. For another example, if the preset parameter scale of the target machine learning model is 1T (Trillion), five growth stages can be preset, the parameter scale corresponding to the first growth stage is 16B, the parameter scale corresponding to the second growth stage is 51B, the parameter scale corresponding to the third growth stage is 100B, the parameter scale corresponding to the fourth growth stage is 500B, and the parameter scale corresponding to the fifth growth stage is 1T.

[0061] For each growth stage in the preset plurality of growth stages, the parameter model expansion of each growth stage can be performed by executing the following steps, and the model after the parameter scale expansion of each growth stage is trained. In this way, by traversing the preset plurality of growth stages, a trained target machine learning model can be obtained, thereby realizing growth training of the target machine learning model through the preset plurality of growth stages.

[0062] Step 102, based on the parameter scale corresponding to the target growth stage, expanding the parameter scale based on the model trained in the previous growth stage of the target growth stage; wherein one or more parameters in the model after expanding the parameter scale are quantized using a low bit width, and the low bit width represents a bit width with a bit number less than a preset value.

[0063] In this step, the parameter scale is expanded for any growth stage in the preset plurality of growth stages; in this way, starting from the first growth stage in the preset plurality of growth stages, the parameter scale is expanded in turn for each growth stage, so as to traverse the preset plurality of growth stages to complete dynamic phased parameter scale expansion; after the expansion of the parameter scale of the last growth stage, the parameter scale of the model reaches the preset parameter scale of the target machine learning model. At the same time, in each growth stage, the parameter scale is expanded on the basis of the model trained in the previous growth stage of the growth stage, so as to inherit the knowledge learned in the previous growth stage, improve the training efficiency, save the training cost, and ensure the performance of the trained target machine learning model.

[0064] Exemplarily, the models after expanding the parameter scale in different growth stages have the same basic structure; the basic structure can be preset as needed, taking the target machine learning model as an example, a large language model, for example, the basic structure of an existing large language model can be used, which can include, for example, an input layer, a transformer architecture, an output layer, etc.

[0065] Exemplarily, the number of network layers, the hidden layer dimension, the number of attention heads, etc. in the target large language model can be preset, and the number of network layers, the hidden layer dimension, and the number of attention heads in the model after expanding the parameter scale in each growth stage do not exceed the preset number of network layers, hidden layer dimension, and number of attention heads; the number of network layers, the hidden layer dimension, and the number of attention heads in the model after expanding the parameter scale in the last growth stage reach the preset number of network layers, hidden layer dimension, and number of attention heads; wherein the number of network layers can represent the number of Transformer blocks, the Transformer block being the core unit of the Transformer architecture, responsible for processing the context information of the input sequence; the hidden layer dimension represents the embedding vector dimension of each token in the large language model, and the hidden layer dimension has an impact on the information representation ability and model complexity of the model; a higher hidden layer dimension can enable the model to represent richer semantic information, because a larger dimension space can accommodate more feature details. The number of attention heads represents the number of attention heads in the Transformer block, the Transformer block adopts a multi-head self-attention mechanism, and the number of attention heads is the number of different "heads" into which the input sequence is divided for self-attention calculation, each attention head can learn different subspace features of the input sequence, so that the model can capture information in the input sequence from multiple angles.

[0066] It should be noted that the first growth stage is an initial growth stage, and there is no previous growth stage, and the trained model corresponding to the previous growth stage can be configured as empty. For example, in the first growth stage, the parameter scale is expanded based on the preset basic structure until the parameter scale corresponding to the first growth stage is reached, so as to complete the parameter scale expansion of the first growth stage; or, in the first growth stage, the preset model can be directly used as the model after the parameter scale expansion of the first growth stage. For example, an open source model or a designed model can be selected as the preset model, and the parameter scale of the preset model is the same as the parameter scale corresponding to the first growth stage.

[0067] In this step, for any growth stage, one or more weight parameters in the model can be quantized using a low bit width after the parameter scale is expanded in the growth stage, so that one or more weight parameters in the model after the parameter scale is expanded are quantized using a low bit width; for example, the low bit width can be used to quantize each weight parameter newly added in the growth stage. The low bit width represents a bit width less than a preset value, and the preset value can be 8 bits, 4 bits, 2 bits, etc.; for example, the low bit width can be 1.58 bits, so as to realize ultra-low bit quantization.

[0068] In one possible implementation, the method further includes: initializing the weight parameters in the newly added attention head and hidden layer dimension to any value in a preset value set, wherein the preset value set includes -1, 0, and 1.

[0069] For example, initializing the weight parameters in the newly added attention head and hidden layer dimension to any value in a preset value set can include: in any growth stage, when the parameter scale of the growth stage is expanded, for any weight parameter in the newly added attention head and the newly added hidden layer dimension that is newly added by the above width expansion, a value in the preset value set {-1, 0, +1} is randomly selected as the initial value of the weight parameter, and the value of the weight parameter only occupies 1.58 bits. In this way, for any growth stage, the weight three-valued processing method is applied to the newly added attention head and hidden layer dimension at the same time, so as to realize quantization of the newly added weight parameter in the model after the parameter scale of the growth stage is expanded using a low bit width, thereby greatly saving the computing power and memory occupation required for training the model in the subsequent growth stage.

[0070] In a possible implementation, the extending the parameter scale based on the model trained in the previous growth stage of the target growth stage comprises: performing two-dimensional parameter scale extension on the model trained in the previous growth stage by width extension and depth extension; the width extension is used to increase the number of attention heads and / or the dimension of the hidden layer in the model; and the depth extension is used to increase the number of network layers of the model. In this way, at each growth stage, the model is extended to the parameter scale corresponding to the growth stage by the two-dimensional extension strategy of width extension and depth extension; and the model scale is gradually extended to the preset parameter scale of the target machine learning model by traversing the preset multiple growth stages.

[0071] For example, the increasing the number of attention heads in the model during the width extension can comprise increasing the number of attention heads in each network layer (such as a Transformer block); and the number of attention heads in different network layers can be the same.

[0072] It should be noted that for any growth stage, the width extension and the depth extension can be performed simultaneously to increase the number of attention heads, the dimension of the hidden layer, and the number of network layers in the model, under the premise that the parameter scale of the extended model reaches the parameter scale corresponding to the growth stage.

[0073] For example, taking the preset parameter scale of the target machine learning model as 101B, setting three growth stages, the parameter scale corresponding to the first growth stage as 16B, the parameter scale corresponding to the second growth stage as 51B, and the parameter scale corresponding to the third growth stage as 101B as an example; in the model after the parameter scale is extended in the first growth stage, the number of network layers can be 64, the number of attention heads can be 64, and the dimension of the hidden layer can be 8192; in the model after the parameter scale is extended in the second growth stage, the number of network layers can be 80, the number of attention heads can be 80, and the dimension of the hidden layer can be 10240; and in the third growth stage, the number of network layers can be 140, the number of attention heads can be 160, and the dimension of the hidden layer can be 20480.

[0074] In a possible implementation, the two-dimensional parameter scale expansion of the model trained in the previous growth stage by width expansion and depth expansion comprises: when the width expansion is performed, the newly added attention heads and hidden layer dimensions in the model trained in the previous growth stage are randomly configured; and / or when the depth expansion is performed, one or more network layers in the model trained in the previous growth stage are selected for replication based on the distance between the input sequence and the output sequence of each network layer in the model trained in the previous growth stage. In this way, at each growth stage, the newly added hidden dimensions and attention heads are randomly initialized in the process of width expansion, to realize dynamic parameter initialization; meanwhile, the number of network layers in the model is increased by replicating the existing network layers in the model in the process of depth expansion, to realize two-dimensional parameter scale expansion of dynamic parameter initialization.

[0075] For example, when the width expansion is performed, the newly added attention heads and hidden layer dimensions in the model trained in the previous growth stage are randomly configured, which can comprise: randomly initializing the newly added attention heads and hidden layer dimensions without exceeding the preset attention heads and hidden dimensions.

[0076] For example, when the depth expansion is performed, one or more network layers in the model trained in the previous growth stage are selected for replication based on the distance between the input sequence and the output sequence of each network layer in the model trained in the previous growth stage, which can comprise: for any growth stage, the distance (such as Euclidean distance, cosine distance, etc.) between the input sequence and the output sequence of each network layer in the model trained in the previous growth stage of the growth stage is calculated, to obtain the distance corresponding to each network layer, and then the network layer with the minimum distance is selected for replication, that is, the network layer with the closest input sequence and output sequence is selected for replication; or the network layers are sorted in ascending order of distance, and multiple network layers are selected for replication; the replicated network layers can be configured at the head or tail position of the model, or at the position adjacent to the original network layer, which is not limited. In this way, based on the input-output distance of each network layer, the network layer with the minimum distance is selected for replication to complete the depth expansion, and the convergence of the model with expanded parameter scale can be improved.

[0077] In step 103, the model with expanded parameter scale is trained, and a first-precision calculation kernel is called to process the operation between the activation matrix and the weight matrix in the training process, to obtain the model trained in the target growth stage; when the model trained in the target growth stage is used to perform an inference task, a second-precision calculation kernel is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

[0078] In the step, in any of the preset growth stages, after the growth stage parameter model is expanded, the model with the expanded parameter scale of the growth stage is trained to obtain a trained model of the growth stage. In this way, the trained target machine learning model is obtained by traversing the preset multiple growth stages, so as to realize the growth training of the target machine learning model through the preset multiple growth stages. It can be understood that, for the first growth stage, if the preset model is directly used as the model with the expanded parameter scale of the first growth stage, then training the model with the expanded parameter scale in the first growth stage is directly training the preset model.

[0079] In a possible implementation, in any of the growth stages, training data for training the model with the expanded parameter scale of the growth stage can be obtained, where the training data can be single-modal data such as text, or multi-modal data such as text, image, and voice, and appropriate training data can be selected based on requirements, which is not limited. For example, the training data can be selected from a public data source. Then, in the process of training the model with the expanded parameter scale of the growth stage by using the training data, the weight matrix and the activation matrix in the model are subjected to matrix multiplication operation based on the probability of the text appearing in the training data, and then the weight values in the weight matrix in the model are updated. Then, after the training in the growth stage reaches a preset termination condition, the training task of the growth stage is completed, and the next growth stage is entered to continue the parameter scale expansion and model training. The preset termination condition can include a preset number of iterations, a preset training time, or model convergence, which is not limited.

[0080] In a possible implementation, the method further includes optimizing the activation value distribution of the copied activation matrix corresponding to the one or more network layers by using a preset transformation. For example, the preset transformation can be Hadamard transformation, which is an orthogonal matrix with elements of ±1. For example, the optimization of the activation value distribution of the copied activation matrix corresponding to the one or more network layers by using the preset transformation can include, in any of the growth stages, when the model with the expanded parameter scale of the growth stage is trained, the activation value distribution of the activation matrix corresponding to the network layer subjected to the deep expansion and copying is optimized to a Gaussian distribution or a Bayesian, Monte Carlo Tree Search (MCTS) distribution, or the like by using Hadamard transformation. For example, the activation matrix corresponding to the network layer can include an activation matrix output by the network layer. In this way, the activation value distribution of the copied network layer is optimized by using Hadamard transformation, and the quantization error is reduced.

[0081] In a possible implementation, the quantization-aware training can be performed in the process of growing training in multiple growth stages; wherein the quantization-aware training (QAT) is a model optimization technique that simulates quantization errors during the training phase, aiming to make the model "adapt in advance" to the low-precision inference environment, thereby significantly reducing the precision loss during deployment. Illustratively, high-precision gradient calculation can be used when back propagation is performed under the condition that the parameter scale of the model is less than a preset threshold; and quantization-aware back propagation is started when the parameter scale of the model expands to more than the preset threshold. In this way, during the growing training process, high-precision gradient calculation is used in the early stage of training to ensure the stability of model convergence, and as the parameter scale of the model expands, quantization-aware back propagation is gradually enabled. For example, taking the preset parameter scale of the target machine learning model as 1T, and presetting 5 growth stages, the parameter scale corresponding to the first growth stage is 16B, the parameter scale corresponding to the second growth stage is 51B, the parameter scale corresponding to the third growth stage is 100B, the parameter scale corresponding to the fourth growth stage is 500B, and the parameter scale corresponding to the fifth growth stage is 1T, FP16 precision can be used to ensure the stability of model convergence when back propagation is performed in the early stage of growing training (such as the first growth stage), and quantization-aware back propagation is enabled when the parameter scale of the model expands to 100B (i.e., the third growth stage).

[0082] Illustratively, the training of the model with the expanded parameter scale includes: in the process of back propagation, the weight matrix of the output layer in the model with the expanded parameter scale is directly calculated by using the least square method, and the weight matrix of the hidden layer in the model with the expanded parameter scale is directly passed through the gradient by using the straight-through estimator (STE). Wherein, the hidden layer can include the network layer between the input layer and the output layer in the model. In this way, for any growth stage using quantization-aware back propagation, the weight matrix of the output layer in the model with the expanded parameter scale in the growth stage is directly calculated by using the least square method in the process of back propagation, avoiding the gradient error accumulation caused by the traditional back propagation method; for example, the feature matrix and the target matrix of the output layer in the model can be directly calculated to obtain the least square solution, and the updated weight of the output layer is obtained. Since it is a one-time algebraic solution, there is no gradient back propagation, thereby avoiding the back propagation gradient error; at the same time, the weight matrix of the hidden layer in the model with the expanded parameter scale in the growth stage is directly passed through the gradient by using the straight-through estimator, bypassing the quantization non-differentiable function, i.e., for the non-differentiable operation (such as round operation, argmax operation, etc.) in the quantization function, the non-differentiable operation is executed in the forward propagation, and the gradient is directly passed through in the back propagation.

[0083] In a possible implementation, the model for each growth stage is calibrated by mixed precision quantization.

[0084] For example, the method further comprises: storing each weight matrix in the model after the extended parameter scale and / or the trained model for the target growth stage by ternarization; and assigning quantization precision of each activation matrix according to the type of each activation matrix in the model after the extended parameter scale and / or the trained model for the target growth stage.

[0085] For example, for any growth stage in the preset plurality of growth stages, after the extended parameter scale for the growth stage is completed, each weight matrix in the model after the extended parameter scale for the growth stage is stored by ternarization, and quantization precision of each activation matrix in the model after the extended parameter scale for the growth stage is assigned according to the type of each activation matrix in the model after the extended parameter scale for the growth stage; for another example, for any growth stage in the preset plurality of growth stages, after the model training for the growth stage is completed, each weight matrix in the trained model for the growth stage is stored by ternarization, and quantization precision of each activation matrix in the trained model for the growth stage is assigned according to the type of each activation matrix in the trained model for the growth stage.

[0086] For example, when each weight matrix is stored by ternarization (-1, 0, 1), the value (i.e., -1, 0, or 1) of each weight in the weight matrix can be stored in a lookup table for quick lookup, where the bit width of the value of each weight is 1.58 bits. For example, each weight matrix in the trained large language model obtained by growth training of the plurality of growth stages is stored by ternarization, which greatly compresses the memory space occupied by the large language model, thereby constructing a lightweight large language model.

[0087] For example, when quantization precision of each activation matrix is assigned, the quantization precision can be selected as needed; for example, the quantization precision can be FP16 (Half Precision Floating Point Number), and the quantized data is 16 bits wide; for another example, the quantization precision can be INT8 (8-bit Integer), and the quantized data is 8 bits wide; for another example, the quantization precision can be INT4 (4-bit Integer), and the quantized data is 4 bits wide.

[0088] Exemplarily, if the type of an activation matrix is an input of an attention layer, i.e., the activation matrix input to the attention layer, the activation matrix can be assigned a quantization precision of a first bit width to quantize the activation values in the activation matrix to the first bit width; if the type of an activation matrix is an intermediate state, such as the activation matrix output by the attention layer, the activation matrix can be assigned a quantization precision of a second bit width to quantize the activation values in the activation matrix to the second bit width, where the first bit width is lower than the second bit width, for example, the first bit width can be 4 bits, and the second bit width can be 8 bits. As an example, 4-bit absolute maximum quantization (absmax) can be performed on the activation matrix input to the attention layer, and 8-bit sparsification can be performed on the activation matrix of the intermediate state. Thus, the quantization precision is dynamically assigned to the activation values in the activation matrix according to the type of the activation matrix.

[0089] Exemplarily, during the model training process, the activation matrix maintains a high precision for any growth stage, and a first-precision calculation kernel is called to process the operation between the activation matrix and the weight matrix; since the quantization precision is assigned to each activation matrix in the trained model, the second-precision calculation kernel is called to process the operation between the activation matrix and the weight matrix when the trained model of the growth stage is used to perform an inference task in the subsequent; the first precision is greater than the second precision; thus, the adaptive scheduling of the calculation kernel is realized, and the hardware scheduling complexity is reduced; the trained model can be deployed on different resources as needed.

[0090] Exemplarily, when the trained model of the growth stage is used to perform an inference task in the subsequent, the mixed-precision general matrix multiplication (mpGEMM) calculation kernel integrated with the ternary lookup table (TL) kernel and the 4 / 8-bit activation dynamic scheduling module can be configured to schedule the calculation kernel of the corresponding precision to perform the operation between the activation matrix and the weight matrix based on the quantization precision (such as 4-bit or 8-bit) assigned to the activation matrix; in this way, resources with calculation kernels of different precisions and smaller memory and display can support the deployment of the model, for example, the trained model can be deployed in an electronic device with 4-bit precision calculation kernels and 8-bit precision calculation kernels to perform an inference task; for the activation matrix with a quantization precision of 8 bits, the 8-bit precision calculation kernel can be scheduled by the 4 / 8-bit activation dynamic scheduling module to perform the operation between the activation matrix and the ternarized weight matrix in combination with the ternary weight lookup table; for the activation matrix with a quantization precision of 4 bits, the 4-bit precision calculation kernel can be scheduled by the 4 / 8-bit activation dynamic scheduling module to perform the operation between the activation matrix and the ternarized weight matrix in combination with the ternary weight lookup table.

[0091] Exemplarily, on the basis of dynamically allocating precision according to types for the activation matrix with ternarization storage of weights, the compiler such as triton and TVM (Tensor Virtual Machine) can be further utilized to ensure the compatibility of the output of the model and / or the trained model after expansion of the parameter scale in each growth stage with the quantized version, that is, to ensure that the quantization parameters corresponding to the same activation matrix before and after expansion of the parameter scale remain consistent, by fusing the function of retaining the mask and the quantization parameter.

[0092] In a possible implementation, the outlier channel reordering is performed for each growth stage.

[0093] Exemplarily, the method further comprises: performing outlier channel detection of the activation matrix; arranging each detected outlier channel at the end of the activation matrix where the outlier channel is located to obtain a reordered activation matrix corresponding to the outlier channel; and synchronously adjusting the positions of the channels in the associated weight matrix based on the positions of the channels in the reordered activation matrix corresponding to the outlier channel, so as to adapt the adjusted weight matrix to the reordered activation matrix corresponding to the outlier channel; and assigning different quantization precisions to the outlier channel and the normal value channel of the activation matrix where the outlier channel is located. In this way, by using the outlier channel reordering technology, the dynamic reordering of the outlier channel of the activation matrix is combined with the static reordering strategy of the weight matrix, and the quantization precision is separately assigned to the outlier channel, so as to effectively reduce the quantization error and improve the model precision; and the inherent contradiction between the model precision and the resource compression is overcome.

[0094] The outlier channel of the activation matrix refers to a channel where an outlier (an abnormal activation value significantly greater than other activation values) of the activation matrix is located.

[0095] Exemplarily, the outlier channel detection of the activation matrix can comprise: detecting the outlier channel of each activation matrix in the model in the process of expanding the parameter scale in each growth stage, or detecting the outlier channel of each activation matrix in the model in the process of training the model after expansion of the parameter scale in each growth stage, or detecting the outlier channel of each activation matrix in the model for the model after expansion of the parameter scale in each growth stage, or detecting the outlier channel of each activation matrix in the model for the trained model in each growth stage.

[0096] Exemplarily, each detected outlier channel is arranged at the end of the activation matrix where the outlier channel is located, to obtain a reordered activation matrix corresponding to the outlier channel; and based on the positions of the channels in the reordered activation matrix corresponding to the outlier channel, the positions of the channels in the associated weight matrix are synchronously adjusted, so that the adjusted weight matrix is adapted to the reordered activation matrix corresponding to the outlier channel; which can include: arranging each detected outlier channel at the end of the activation matrix where the outlier channel is located, to dynamically reorder the outlier channels, and at the same time, for the weight matrix (i.e. the associated weight matrix) that needs to perform matrix multiplication operation with the activation matrix where the outlier channel is located, synchronously performing static reordering of the channels, to adapt to the channels in the reordered activation matrix; in this way, when the reordered activation matrix and the reordered associated weight matrix perform matrix multiplication operation, the correspondence between the channels is consistent with the correspondence between the channels when the pre-reordered activation matrix and the pre-reordered associated weight matrix perform matrix multiplication operation, thereby ensuring the computational equivalence of matrix multiplication.

[0097] Exemplarily, assigning different quantization precisions to the outlier channel and the normal value channels of the activation matrix where the outlier channel is located can include: assigning a higher quantization precision to the outlier channel than to the normal value channels; in this way, considering the influence of the outlier value on the model performance, a higher quantization precision is assigned to the outlier channel in the activation matrix, thereby effectively reducing the quantization error.

[0098] As an example, for the activation matrix of the input attention layer, if it is detected that there is an outlier channel in the activation matrix, the outlier channel is assigned a first bit width quantization precision to quantize the activation values in the outlier channel to the first bit width, and at the same time, the other channels (i.e. normal value channels) in the activation matrix are assigned a second bit width quantization precision to quantize the normal value channels to the second bit width, wherein the first bit width is greater than the second bit width, for example, the first bit width can be 4 bits and the second bit width can be 8 bits. In this way, during the growing training process, the outlier channel reordering technology is used to dynamically reorder the outlier channels in the activation matrix to concentrate the outlier channels at the end of the activation matrix, and 8-bit quantization precision is assigned to the outlier channels, while 4-bit quantization precision is assigned to the other normal value channels, and the associated weight matrix is synchronously executed static reordering to ensure the computational equivalence of matrix multiplication; compared to using 4-bit quantization for each channel (including outlier channels and normal value channels) in the activation matrix, the quantization error can be reduced by 42%. At the same time, through mathematical reasoning task (GSM8K, Grade School Math 8K) evaluation, the accuracy rate of using the outlier channel reordering technology still remains at 58.38%, which is 1.2 percentage points higher than that of the full-precision model of the same scale.

[0099] In a possible implementation, on the basis of performing the abnormal channel reordering, a residual approximation strategy can be further adopted, the residual approximation strategy takes the output of the previous growth stage as a teacher signal, and through weighted fusion of language modeling loss and quantization error loss, the performance difference of the model after scale expansion and fine-tuning is less than 0.1%, which is significantly better than the 15% precision loss of the traditional post-training quantization method.

[0100] In the embodiments of the present disclosure, through steps 101-103, the parameter scale corresponding to the target growth stage is obtained; wherein the target growth stage is any growth stage in a plurality of preset growth stages, the parameter scale corresponding to each growth stage in the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; the plurality of growth stages are used to train the target machine learning model by growth training; based on the parameter scale corresponding to the target growth stage, the parameter scale of the model trained in the previous growth stage of the target growth stage is expanded; wherein one or more parameters in the model after scale expansion are quantized by a low bit width, and the low bit width represents a bit width with a bit number less than a preset value; the model after scale expansion is trained, and a first precision calculation core is called to process the operation between the activation matrix and the weight matrix in the training process to obtain the model trained in the target growth stage; wherein when the model trained in the target growth stage is used to execute an inference task, a second precision calculation core is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision. In this way, by fusing growth parameter expansion and low bit width quantization, the memory and video memory occupation in the training process is significantly reduced, the training efficiency is improved, the cost of machine learning model training is saved, and efficient training of the machine learning model is realized. At the same time, the parameters in the model are quantized by a low bit width, which greatly reduces the memory and video memory occupied by the model and reduces the limitation on the deployment resources; and the first precision calculation core is called to process the operation between the activation matrix and the weight matrix in the training stage, and the second precision calculation core is called to process the operation between the activation matrix and the weight matrix in the inference stage, thereby realizing adaptive scheduling of the calculation core and reducing the hardware scheduling complexity; the trained model can be deployed on different resources as needed, for example, a resource with a calculation core of different precision and small memory and video memory can support the deployment of the model.

[0101] Among them, the growing parameter expansion, also known as dynamic parameter expansion or low-rank parameter expansion, as a large model training method to reduce the computational requirements, reduces the computational amount and memory usage of large model training, and further reduces the hardware and training cost required for training. It has been verified that the total cost of training a model with a trillion parameters using the above method in the embodiments of the present disclosure is reduced to the level of 100,000 US dollars, which is more than 90% less than the traditional method, and the memory usage is reduced by 95% (for example, a 2B parameter model only needs 0.4GB).

[0102] Figure 2 A schematic diagram of training loss of a growing training model according to an embodiment of the present disclosure is shown, as Figure 2 shown, divided into three growth stages, namely growth stage 1, growth stage 2, and growth stage 3; wherein the parameter size corresponding to the growth stage 1 is 16B; the parameter size corresponding to the growth stage 2 is 51B; the parameter size corresponding to the growth stage 3 is 101B. Using the above training method in the embodiments of the present disclosure, through the growth stage 1, the growth stage 2, and the growth stage 3, the model with a parameter size of 101B is gradually trained from the parameter size of 16B, wherein Figure 2 The trend of training loss (Training Loss) with the number of processed tokens (Processed Tokens) is shown in the figure, it can be seen that during the training process, the training loss continues to decrease rapidly, and a small number of tokens can achieve ideal training effect, improve the training efficiency and save the training cost.

[0103] Figure 3(a) shows a schematic diagram of the change curve of model training loss under different hyperparameter combinations according to an embodiment of the present disclosure, as shown in Figure 3(a), different hyperparameter combinations are searched using the grid search method, and the above method is used to train based on different hyperparameter combinations. In the figure, the trend of model training loss (Training Loss) with training steps (Steps) can be seen, wherein different colors represent different hyperparameter combinations; it can be seen that the training loss continues to decrease based on different hyperparameter combinations using the above method in the embodiments of the present disclosure, and the loss function can converge quickly, achieving ideal learning effect.

[0104] Figure 3(b) shows a schematic diagram of the gradient vector norm variation curves under different hyperparameter combinations according to an embodiment of the present disclosure. As shown in Figure 3(b): different hyperparameter combinations are searched using a grid search method, and the above-mentioned method in the embodiment of the present disclosure is used for training based on different hyperparameter combinations. The gradient vector norm can be seen to change with training deployment, where different colors represent different hyperparameter combinations. It can be seen that by using the above-mentioned method in the embodiment of the present disclosure for training based on different hyperparameter combinations, satisfactory results can be achieved in terms of gradient "magnitude" and "stability" after a relatively short number of training steps.

[0105] Figure 4 This diagram illustrates the effect of low-bit quantization according to an embodiment of the present disclosure, as shown below. Figure 4 As shown, for large language models with the same parameter size, when training the large language model to the same performance, the training cost of the low-bit quantization large language model obtained using the training method in this embodiment is significantly lower than the training cost of the traditional large language model using FP16 precision. Furthermore, because this embodiment uses low-bit quantization, the difficulty of multiplication operations between the weight matrix and activation matrix during inference is reduced, thereby lowering the resource requirements for model deployment. Traditional FP16 precision large language models typically require deployment on GPUs, while the large language model obtained using the training method in this embodiment can also be deployed on CPUs with x86 / ARM architectures.

[0106] For example, for a large language model with a preset parameter size of 1T, the method described above in this embodiment is used for training. Through width-depth dual-dimensional expansion at each growth stage, the model gradually grows from a 16B parameter size to a 1T parameter size. During width expansion, the newly added hidden layer dimensions and attention heads are randomly initialized, and weight trivaluation preprocessing is applied simultaneously. During depth expansion, the layer with the closest input-output distance is selected for replication, and the replicated parameters are optimized for activation distribution using Hadamard transformation to reduce quantization error. Thus, the model starts with a 16B parameter size and gradually grows to a 1T size through width expansion (random initialization of newly added hidden layer dimensions) and depth expansion (layer replication based on input-output distance). Each growth stage embeds weight trivaluation preprocessing ({-1, 0, +1}, 1.58 bits / parameter), combined with Hadamard transformation to optimize activation distribution, reducing quantization error by 42%. Furthermore, after each growth stage is completed, the model is calibrated using mixed-precision quantization: weights are stored in ternary form (1.58 bits), and activation values ​​are dynamically assigned precision according to the layer type (attention layer inputs use 4-bit absmax quantization, and intermediate states use 8-bit sparsity); the mask and quantization parameters are preserved through the fusion function to ensure compatibility between the expanded model output and the quantized version.

[0107] Further, after completing the growth training of the preset plurality of growth stages, the weights and quantization parameters of the trained target machine learning model can be packaged in a preset format; for example, the weights and quantization parameters of the trained target machine learning model can be uniformly packaged in a GGUF (GPT-Generated Unified Format) format; wherein the GGUF is a binary file format designed for large language models, and its core value lies in uniformly packaging the original model weights and quantization parameters into a single file to achieve efficient deployment and cross-platform compatibility. Further, the packaged model can be deployed on different platforms; for example, it can support full-platform deployment from platforms that originally perform 4-bit calculations to Apple M series chips, improving the hardware ecosystem compatibility of the model. It has been verified that in long text processing scenarios, using low-bit quantization combined with RoPE and other post-quantization technologies reduces the memory occupancy of 16K context by 80%, and the inference delay of edge devices is stable within 120ms. This integrated training-inference design breaks through the contradiction between the expansion law and the growth rate of hardware resources in traditional frameworks, providing a feasible path for the landing of trillion-parameter models on consumer-grade devices.

[0108] In addition, the disclosure also provides a mixed-precision general matrix multiplication (mpGEMM) calculation core. A unified mixed-precision calculation core (mpGEMM) can be developed using existing tools, integrating a ternary weight lookup table (TL (Ternary Lookup) core) and a 4 / 8-bit activation dynamic scheduling module. When growth training is performed through multiple growth stages, the FP16 core is preferentially called to complete model parameter scale expansion, and in the inference stage, the low-bit quantization core can be automatically switched to; thereby realizing hardware adaptive scheduling and training and inference collaboration. In the inference stage, the mixed-precision calculation core realizes a 6.46-fold speed improvement under x86 / ARM architecture, and a single CPU can achieve a human reading level response speed of 5-7 tokens / second. In certain parameter scales, when deployed on chips such as Apple M2 for inference, the energy consumption is as low as 0.028 joules / token.

[0109] FIG. 5(a) shows a comparison diagram of inference speed of different models deployed on a certain chip according to an embodiment of the present disclosure. As shown in FIG. 5(a), the low-bit quantized model trained by the method described above in the embodiments of the present disclosure is deployed on an Apple M2 chip, and the inference speed is compared with that of a control model (such as llama.cpp) quantized by FP16 or INT4 and deployed on the Apple M2 chip. Among them, the human reading speed is 5-7 tokens / s as a benchmark. Under the condition that the model parameter scale is the same, the inference speed of the low-bit quantized model trained by the method described above in the embodiments of the present disclosure is significantly faster than that of the control model. At the same time, when the model parameter scale reaches 100B, the inference speed of the control model is significantly lower than the human reading speed, while the inference speed of the low-bit quantized model trained by the method described above in the embodiments of the present disclosure can still match the human reading speed.

[0110] FIG. 5(b) shows a comparison diagram of inference energy consumption of different models deployed on a certain chip according to an embodiment of the present disclosure. As shown in FIG. 5(b), the low-bit quantized model trained by the method described above in the embodiments of the present disclosure is deployed on an Apple M2 chip, and the inference energy consumption is compared with that of a control model (such as llama.cpp) quantized by FP16 or INT4 and deployed on the Apple M2 chip. Among them, under the condition that the parameter scale of the model is 700M, the energy consumption of the low-bit quantized model trained by the method described above in the embodiments of the present disclosure is reduced by 55.4% compared with that of the control model; under the condition that the parameter scale of the model is 70B, the energy consumption of the low-bit quantized model trained by the method described above in the embodiments of the present disclosure is reduced by 70% compared with that of the control model; it can be seen that the model trained by the method described above in the embodiments of the present disclosure can greatly save energy consumption.

[0111] In this way, in some scenarios, during the training process of combining the growing parameter expansion and the low-bit width quantization, the dynamic parameter initialization, the quantization-aware training, and the mixed precision can be combined, thereby achieving a breakthrough balance between training efficiency, resource compression, and model performance, and achieving large-scale computing power saving for large model training and inference. The following contradictions faced by existing training of light large language models are solved: positive correlation between training cost and model scale: existing scaling laws require that the parameter scale and the data volume grow synchronously, but the growth rate of hardware resources lags behind the demand for model complexity improvement; antagonism between model accuracy maintenance and resource compression: the quantization technology generally faces the "compression rate-accuracy-energy consumption" impossible triangle, and needs to achieve bit-level calculation reconstruction through structural innovation; the training function retention mechanism relies on static masks, which is difficult to adapt to dynamic topology adjustment of trillion-level parameters.

[0112] Based on the same inventive concept of the method embodiments, the embodiments of the present disclosure also provide a device for training a model with low bit width and dynamic parameter initialization. The device can be used to implement the technical solutions described in the above method embodiments.

[0113] Figure 6 FIG. 6 shows a structural diagram of a device for training a model with low bit width and dynamic parameter initialization according to an embodiment of the present disclosure. Figure 6 As shown in FIG. 6, the device includes: an acquisition module 601 configured to acquire a parameter scale corresponding to a target growth stage; wherein the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each growth stage of the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth training; an expansion module 602 configured to expand the parameter scale based on the parameter scale corresponding to the target growth stage on the basis of a model trained in a previous growth stage of the target growth stage; wherein one or more parameters in the model after the parameter scale is expanded are quantized using a low bit width, and the low bit width represents a bit width with a bit number less than a preset value; and a training module 603 configured to train the model after the parameter scale is expanded, and to call a first precision calculation kernel to process operations between an activation matrix and a weight matrix during the training to obtain a model trained in the target growth stage; wherein the model trained in the target growth stage is configured to call a second precision calculation kernel to process operations between the activation matrix and the weight matrix when the model is used to perform an inference task; and the first precision is greater than the second precision.

[0114] In the embodiments of the present disclosure, a parameter scale corresponding to a target growth stage is acquired, wherein the target growth stage is any growth stage in a plurality of preset growth stages, the parameter scale corresponding to each growth stage in the plurality of growth stages is determined based on a preset parameter scale of a target machine learning model, the plurality of growth stages are used to train the target machine learning model through growth training, the parameter scale is expanded based on the parameter scale corresponding to the target growth stage and a model trained in a previous growth stage of the target growth stage, wherein one or more parameters in the model after the parameter scale is expanded are quantized by using a low bit width, the low bit width represents a bit width with a bit number less than a preset value, the model after the parameter scale is expanded is trained, and a first precision calculation core is called to process an operation between an activation matrix and a weight matrix in a training process to obtain a model trained in the target growth stage, wherein the model trained in the target growth stage is used to call a second precision calculation core to process an operation between the activation matrix and the weight matrix when performing an inference task, and the first precision is greater than the second precision. In this way, by fusing growth parameter expansion and low bit width quantization, the memory and the display memory occupied in the training process are significantly reduced, the training efficiency is improved, the cost of machine learning model training is saved, and efficient training of the machine learning model is realized. At the same time, the parameters in the model are quantized by using the low bit width, the memory and the display memory occupied by the model are greatly reduced, and the limitation on deployment resources is reduced. Moreover, the first precision calculation core is called to process the operation between the activation matrix and the weight matrix in the training stage, and the second precision calculation core is called to process the operation between the activation matrix and the weight matrix in the inference stage, so that adaptive scheduling of the calculation core is realized, and the hardware scheduling complexity is reduced. The model after the training can be deployed on different resources as needed, for example, resources with different precision calculation cores and small memory and display memory can support deployment of the model.

[0115] In a possible implementation, the expansion module 602 is further configured to perform two-dimensional parameter scale expansion on the model trained in the previous growth stage by width expansion and depth expansion, wherein the width expansion is used to increase the number of attention heads and / or the dimension of the hidden layer in the model, and the depth expansion is used to increase the number of network layers in the model.

[0116] In a possible implementation, the expansion module 602 is further configured to randomly configure the number of newly added attention heads and the dimension of the hidden layer in the model trained in the previous growth stage when performing the width expansion, and / or select one or more network layers in the model trained in the previous growth stage to be copied based on the distance between the input sequence and the output sequence corresponding to each network layer in the model trained in the previous growth stage when performing the depth expansion.

[0117] In a possible implementation, the expansion module 602 is further configured to: initialize the weight parameters in the added attention head and the hidden layer dimension to any value in a preset value set, where the preset value set includes three values: -1, 0, and 1; and / or optimize the activation value distribution of the activation matrix corresponding to the copied one or more network layers by using a preset transformation.

[0118] In a possible implementation, the training module 603 is further configured to: store each weight matrix in the model after the expansion of the parameter scale and / or the model trained for the target growth stage by using ternarization; and assign a quantization precision of each activation matrix according to a type of the activation matrix in the model after the expansion of the parameter scale and / or the model trained for the target growth stage.

[0119] In a possible implementation, the training module 603 is further configured to: perform outlier channel detection on the activation matrix; arrange each detected outlier channel at an end of the activation matrix where the outlier channel is located, to obtain a reordered activation matrix corresponding to the outlier channel; and based on positions of each channel in the reordered activation matrix corresponding to the outlier channel, synchronously adjust positions of each channel in an associated weight matrix, so that the adjusted weight matrix is adapted to the reordered activation matrix corresponding to the outlier channel; and assign different quantization precisions to the outlier channel and normal value channels of the activation matrix where the outlier channel is located.

[0120] In a possible implementation, the training module 603 is further configured to: in the process of back propagation, directly calculate the weight of the weight matrix of the output layer in the model after the expansion of the parameter scale by using the least square method, and pass the gradient by using a straight-through estimator for the weight matrix of the hidden layer in the model after the expansion of the parameter scale.

[0121] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For briefness, details are not described herein again.

[0122] The embodiments of the present disclosure further provide an electronic device, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0123] The embodiments of the present disclosure further provide a non-volatile computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0124] The embodiment of the present disclosure further provides a computer program product, comprising a computer program or a nonvolatile computer readable storage medium carrying the computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0125] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0126] The electronic device 1900 can also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0127] In an exemplary embodiment, a non-volatile computer readable storage medium, such as the memory 1932 including computer program instructions, is also provided, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0128] Computer readable storage media can be any media that can be read by a machine. Such media can include, but is not limited to, optical discs, magnetic discs, magnetic tapes, electronic memories, and / or any combination thereof. Computer readable storage media can be non-transitory, in that it can be a tangible medium. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se, but can be a tangible medium. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal. In some embodiments, computer readable storage media can be non-transitory, in that it can not be a signal per se, but can be a tangible medium. In other embodiments, computer readable storage media can be a transitory medium, in that it can be a signal.

[0129] The computer programs (or computer readable program instructions) described herein can be downloaded from a computer readable storage medium to respective computing / processing devices or to external computers or external storage devices via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0130] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0131] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0132] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0133] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0134] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0135] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the described embodiments are possible, and all such modifications and variations are intended to be within the scope of the described embodiments. The description used herein is intended to best explain the principles of the various embodiments, the practical application, and the best mode of using the present disclosure, and to enable others skilled in the art to understand the disclosure, various embodiments, and the application, devices, and apparatuses.

Claims

1. A method for initializing a model training process using low bit width and dynamic parameters, characterized in that, The method includes: Obtain the parameter scale corresponding to the target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth. Based on the parameter scale corresponding to the target growth stage, the parameter scale is expanded on the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the expanded parameter scale model are quantized with low bit width, and the low bit width means that the number of bits is less than the preset value of the bit width. The model with the expanded parameter scale is trained, and during the training process, a first-precision computation kernel is called to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision computation kernel is called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

2. The method according to claim 1, characterized in that, Expanding the parameter scale based on the model trained in the previous growth stage of the target growth stage includes: The model trained in the previous growth stage is expanded in two dimensions by width expansion and depth expansion; wherein, the width expansion is used to increase the number of attention heads and / or the hidden layer dimension in the model; and the depth expansion is used to increase the number of network layers in the model.

3. The method according to claim 2, characterized in that, The process of expanding the parameter scale of the model trained in the previous growth stage through width and depth expansion includes: When performing the width expansion, the number of new attention heads and the hidden layer dimension are randomly configured in the model trained in the previous growth stage. And / or, During the depth expansion, based on the distance between the input and output sequences of each network layer in the model trained in the previous growth stage, one or more network layers in the model trained in the previous growth stage are selected for replication.

4. The method according to claim 3, characterized in that, The method further includes: The newly added attention head and the weight parameters in the hidden layer dimension are initialized to any value in a preset set of values, wherein the preset set of values ​​includes three values: -1, 0, and 1. And / or, The activation value distribution of the activation matrix corresponding to the one or more network layers is optimized through a preset transformation.

5. The method according to claim 4, characterized in that, The method further includes: The weight matrices in the model after the parameter scaling is ternary and / or in the model trained during the target growth stage are stored using ternary representation. The quantization precision of each activation matrix is ​​assigned based on the type of each activation matrix in the model after the expansion parameter scale and / or in the model trained in the target growth stage.

6. The method according to claim 4 or 5, characterized in that, The method further includes: Perform outlier channel detection on the activation matrix; Each detected outlier channel is placed at the end of the activation matrix where the outlier channel is located, resulting in the reordered activation matrix corresponding to the outlier channel. Based on the position of each channel in the reordered activation matrix corresponding to the outlier channel, the position of each channel in the associated weight matrix is ​​adjusted synchronously so that the adjusted weight matrix is ​​adapted to the reordered activation matrix corresponding to the outlier channel. Assign different quantization precisions to the outlier channel and the normal value channel in the activation matrix containing the outlier channel.

7. The method according to claim 1, characterized in that, Training the model after expanding the parameter scale includes: During backpropagation, the weights of the output layer in the model with the expanded parameter size are directly calculated using the least squares method, and the gradients of the hidden layer in the model with the expanded parameter size are passed through an estimator.

8. A model training device that uses low bit width and dynamic parameter initialization, characterized in that, The device includes: An acquisition module is used to acquire the parameter scale corresponding to a target growth stage; wherein, the target growth stage is any one of a plurality of preset growth stages, and the parameter scale corresponding to each of the plurality of growth stages is determined based on the preset parameter scale of the target machine learning model; the plurality of growth stages are used to train the target machine learning model through growth. The extension module is used to expand the parameter scale based on the parameter scale corresponding to the target growth stage and the model trained in the previous growth stage of the target growth stage; wherein, one or more parameters in the model after expanding the parameter scale are quantized with low bit width, and the low bit width means that the number of bits is less than a preset value. The training module is used to train the model after expanding the parameter scale, and during the training process, it calls a first-precision calculation kernel to process the operation between the activation matrix and the weight matrix to obtain the model trained in the target growth stage; wherein, when the model trained in the target growth stage is used to perform inference tasks, a second-precision calculation kernel will be called to process the operation between the activation matrix and the weight matrix; the first precision is greater than the second precision.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images