Quantification method and device, electronic device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a quantification method and device, an electronic device, a storage medium and electronic equipment. The method comprises the following steps: obtaining a rotation matrix for executing rotation operation, wherein the rotation operation comprises left rotation operation and right rotation operation; obtaining a weight matrix of a plurality of layers of the artificial intelligence model; respectively performing one of a left rotation operation and a right rotation operation on the weight matrixes of the plurality of layers by using the rotation matrix according to the types of the plurality of layers; and performing quantization operation on the weight matrixes of the plurality of layers after the rotation operation is performed. According to the method, mathematical equivalence quantitative operation can be realized aiming at an artificial intelligence model with a highly dissimilatory structure with low calculation complexity.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence, and particularly to quantization methods, electronic devices, storage media, electronic devices, and quantization apparatuses. Background Technology

[0002] Recently, AI models such as Large Language Models (LLMs) have been widely used, and these models may have a large number of parameters. Quantization processes can be performed on the parameters of AI models to reduce their size, memory footprint, inference speed, and power consumption, enabling them to run on resource-constrained edge devices.

[0003] Furthermore, the parameters of artificial intelligence models may contain a large number of outliers. If the parameters are directly quantized, the outliers may crowd out the limited precision space after quantization, leading to a severe loss of accuracy. A better approach is to first perform a mathematical rotation on the parameters to make their distribution closer to isotropic, thereby improving the fidelity of subsequent quantization operations.

[0004] However, the aforementioned rotation and quantization operations are suitable for highly regular AI models. When these operations are directly applied to highly heterogeneous AI models, it leads to a problem where the AI ​​models before and after quantization are mathematically inequivalent. Summary of the Invention

[0005] According to at least one embodiment of the present disclosure, a rotation-based quantization method for an artificial intelligence model is provided, comprising: obtaining a rotation matrix for performing rotation operations, wherein the rotation operations include a left rotation operation and a right rotation operation; obtaining weight matrices of multiple layers of the artificial intelligence model; performing one of a left rotation operation and a right rotation operation on the weight matrices of the multiple layers respectively using the rotation matrix, depending on the type of the multiple layers; and performing a quantization operation on the weight matrices of the multiple layers after performing the rotation operations.

[0006] For example, the method according to at least one embodiment of the present disclosure further includes: dividing the obtained artificial intelligence model into multiple slices; using multiple computing nodes to perform rotation and quantization operations on the weight matrices of multiple layers of the divided multiple slices, respectively, wherein the multiple computing nodes use the same rotation matrix; and aggregating the multiple slices obtained by performing rotation and quantization operations using the multiple computing nodes into the artificial intelligence model after performing rotation and quantization operations.

[0007] For example, according to a method of at least one embodiment of the present disclosure, wherein the rotation matrix includes a first rotation matrix and a second rotation matrix, the method further includes: generating a third rotation matrix as a random Hadamard matrix by a first process running on a first computing node among a plurality of computing nodes; calculating a rotation scaling factor of the third rotation matrix and a fourth rotation matrix as the inverse of the third rotation matrix by the first process; dividing the third rotation matrix by the rotation scaling factor to obtain the first rotation matrix, and dividing the fourth rotation matrix by the rotation scaling factor to obtain the second rotation matrix; and transmitting one or more of the first rotation matrix, the second rotation matrix, the third rotation matrix, the fourth rotation matrix, and the rotation scaling factor by the first process to remaining processes running on remaining computing nodes other than the first computing node among the plurality of computing nodes.

[0008] For example, according to at least one embodiment of the method of this disclosure, the step of performing a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers respectively using the rotation matrix according to the type of the plurality of layers further includes: performing a left rotation operation on the weight matrix of at least one of the query projection layer, key / value projection layer, dimensionality-up projection layer, gated projection layer, and router projection layer of the plurality of layers using the rotation matrix; and performing a right rotation operation on the weight matrix of at least one of the output projection layer and dimensionality-down projection layer of the plurality of layers using the rotation matrix.

[0009] For example, according to at least one embodiment of the method of this disclosure, the rotation matrix includes a first rotation matrix and a second rotation matrix that is the inverse of the first rotation matrix, and one of the steps of performing a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix further includes: multiplying the weight matrix of a first portion of the plurality of layers on the left using the second rotation matrix to perform a left rotation operation; and multiplying the weight matrix of a second portion of the plurality of layers on the right using the first rotation matrix to perform a right rotation operation.

[0010] For example, according to at least one embodiment of the method of this disclosure, wherein the plurality of layers includes a root mean square normalization layer and a linear projection layer disposed immediately after the root mean square normalization layer, the method further includes: fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer; and setting the weights in the weight matrix of the root mean square normalization layer to one.

[0011] For example, according to at least one embodiment of the method of this disclosure, the step of fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer immediately following the root mean square normalization layer includes: left-multiplying the weight matrix of the linear projection layer immediately following the root mean square normalization layer by the weight matrix of the root mean square normalization layer to obtain the fused weight matrix of the linear projection layer immediately following the root mean square normalization layer.

[0012] For example, according to a method of at least one embodiment of this disclosure, the root mean square normalization layer includes at least one of an input root mean square normalization layer and a post-attention root mean square normalization layer; and the linear projection layer immediately following the root mean square normalization layer includes at least one of a query projection layer, a key / value projection layer, an upscaling projection layer, a gated projection layer, and a router projection layer.

[0013] For example, according to at least one embodiment of the method of this disclosure, the fusion of the weight matrix of the root mean square normalized layer into the weight matrix of the linear projection layer immediately following the root mean square normalized layer is performed before performing one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix; and wherein performing one of the left rotation operation and right rotation operation on the weight matrices of the plurality of layers using the rotation matrix includes performing at least one of a left rotation operation and a right rotation operation on the weight matrix of the linear projection layer immediately following the root mean square normalized layer after the fusion operation using the rotation matrix.

[0014] For example, according to a method of at least one embodiment of the present disclosure, wherein the plurality of layers include an embedding layer, and the method further includes: performing a centering operation on the weights in the weight matrix of the embedding layer; and performing a right rotation operation on the weight matrix of the embedding layer, which has undergone the centering operation, using the rotation matrix.

[0015] For example, according to a method of at least one embodiment of the present disclosure, performing a centering operation on the weights in the weight matrix of the embedding layer includes: determining the mean of the weights in each row of the weight matrix of the embedding layer; and subtracting the mean of the row in which the weight is located from each weight in the weight matrix of the embedding layer to perform a centering operation.

[0016] For example, according to at least one embodiment of the method of this disclosure, the plurality of layers include an output head layer, and one of the steps of performing a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix further includes: performing a left rotation operation on the weight matrix of the output head layer using the rotation matrix.

[0017] For example, according to at least one embodiment of the method of this disclosure, wherein the plurality of layers include a model normalization layer, and the output head layer is set immediately after the model normalization layer, the method further includes: fusing the weight matrix of the model normalization layer into the weight matrix of the output head layer set immediately after the model normalization layer; and setting the weights in the weight matrix of the model normalization layer to one, wherein performing a left rotation operation on the weight matrix of the output head layer using the rotation matrix includes: performing a left rotation operation on the weight matrix of the output head layer after the fusion operation using the rotation matrix.

[0018] For example, according to at least one embodiment of the method of this disclosure, obtaining the weight matrix of the multiple layers of the artificial intelligence model includes: obtaining the weight matrix of the multiple layers of the artificial intelligence model with a precision not lower than a predetermined value.

[0019] For example, according to at least one embodiment of the method of this disclosure, obtaining a weight matrix of multiple layers of the artificial intelligence model with a precision not lower than a predetermined precision includes: obtaining a weight matrix of multiple layers of the artificial intelligence model with a first precision and a corresponding quantization scaling factor, wherein the first precision is lower than the predetermined precision; performing dequantization on the weight matrix of the first precision at least based on the corresponding quantization scaling factor to obtain the weight matrix with a precision not lower than the predetermined precision.

[0020] For example, according to at least one embodiment of the method of this disclosure, the step of performing a quantization operation on the weight matrix of the multiple layers after performing a rotation operation includes: performing a quantization operation on the weight matrix of the multiple layers after performing a rotation operation to obtain a weight matrix with a second precision, wherein the second precision is lower than the first precision.

[0021] For example, according to a method of at least one embodiment of this disclosure, the artificial intelligence model includes a large hybrid expert architecture model.

[0022] For example, according to at least one embodiment of the method of this disclosure, the weight matrix of the multiple layers after performing the quantization operation satisfies one or more of the following: the memory usage of the weight matrix of the multiple layers after performing the quantization operation is lower than the memory usage of the weight matrix of the multiple layers before performing the quantization operation; the transmission bandwidth of the weight matrix of the multiple layers after performing the quantization operation is lower than the transmission bandwidth of the weight matrix of the multiple layers before performing the quantization operation; and the running speed of the artificial intelligence model having the weight matrix of the multiple layers after performing the quantization operation is higher than the running speed of the artificial intelligence model having the weight matrix of the multiple layers before performing the quantization operation.

[0023] An electronic device is provided according to at least one embodiment of the present disclosure, comprising: at least one processing unit; at least one memory including one or more computer program modules, wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processing unit, the one or more computer program modules being used to implement the method described above.

[0024] According to at least one embodiment of the present disclosure, a computer-readable storage medium is provided having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described above.

[0025] According to at least one embodiment of the present disclosure, a rotation-based quantization apparatus for an artificial intelligence model is provided, comprising: a rotation matrix obtaining module configured to obtain a rotation matrix for performing a rotation operation, wherein the rotation operation includes a left rotation operation and a right rotation operation; a weight matrix obtaining module configured to obtain weight matrices of multiple layers of the artificial intelligence model; a rotation module configured to perform one of a left rotation operation and a right rotation operation on the weight matrices of the multiple layers respectively, based on the type of the multiple layers; and a quantization module configured to perform a quantization operation on the weight matrices of the multiple layers after performing the rotation operation.

[0026] An electronic device is provided according to at least one embodiment of the present disclosure, including an electronic device of the embodiment described above or a quantization device of the embodiment described above.

[0027] According to at least one embodiment of the method disclosed herein, in the rotation-based quantization process for large language models (e.g., MoE architecture large models) applied to natural language processing and reasoning, the accuracy of artificial intelligence models can be maintained under low-bit (e.g., FP8, INT4) quantization conditions, while ensuring mathematical equivalence and deployment compatibility.

[0028] The method according to at least one embodiment of this disclosure fully considers the complex and highly heterogeneous structure in the MoE architecture model, and designs detailed and unified rotation operations for router networks, shared expert networks, and a large number of independent expert networks in the MoE architecture model, thereby ensuring mathematical equivalence.

[0029] Furthermore, the method according to at least one embodiment of this disclosure performs only the aforementioned simplified rotation operation on each layer, without performing further rotation operations on specific layers (e.g., Q-layer, K / V-layer, O-layer, and Down-layer). In this way, by reducing the number of rotation operations with low returns based on the fact that the weight matrices of each layer tend to be isotropic, computational overhead can be reduced. This is especially important when the hidden layer dimension is large and the number of layers is extremely deep.

[0030] The method according to at least one embodiment of this disclosure achieves globally mathematically equivalent low-bit quantization in the MoE architecture by performing rotation, centering, and fusion operations on the layers of the artificial intelligence model, thereby systematically solving the application problem of rotation-based quantization operations on complex MoE models. Furthermore, by performing a fusion operation on the weight matrices of the root mean square normalized layer and the model normalized layer, modification of the inference code can be avoided, thus reducing the complexity of use. Attached Figure Description

[0031] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A flowchart of a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown.

[0033] Figure 2 A schematic diagram of a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown.

[0034] Figure 3 A schematic diagram of a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown.

[0035] Figure 4 A schematic structural diagram of an artificial intelligence processor according to at least one embodiment of the present disclosure is shown.

[0036] Figure 5 A schematic diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0037] Figure 6 A schematic diagram of a quantization apparatus according to at least one embodiment of the present disclosure is shown.

[0038] Figure 7 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown. Detailed Implementation

[0039] Before proceeding with the detailed description below, it may be advantageous to define certain words and phrases used throughout this disclosure. The terms “comprising” and “including” and their derivatives mean including but not limited to. The term “or” is inclusive, meaning and / or. The phrase “associated with” and its derivatives mean including, comprising, interconnecting, containing, contained within, connected or connected to, coupled or coupled to, communicating with, cooperating, intertwining, juxtaposing, proximate, binding or bound to, having, possessing attributes, having a relationship or being related to, etc. The term “controller” means any device, system, or part thereof that controls at least one operation. Such a controller may be implemented in hardware, or a combination of hardware and software and / or firmware. The functionality associated with any particular controller may be centralized or distributed, local or remote. The phrase “at least one,” when used with a list of items, means that different combinations of one or more of the listed items may be used, and that only one item from the list may be required. For example, "at least one of A, B, and C" includes any one of the following combinations: A, B, C, A and B, A and C, B and C, A and B and C.

[0040] Definitions of other specific words and phrases are provided throughout this disclosure. Those skilled in the art will understand that, in many, if not most, cases, such definitions apply to the prior and future use of the words and phrases thus defined.

[0041] The various embodiments of the principles of this disclosure described below with reference to the accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this disclosure in any way. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged system or device. In some cases, the actions described in the specification may be performed in a different order and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily require a specific order or sequential sequence to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0042] Recently, artificial intelligence models such as Large Language Models (LLMs) have been widely used, and these models may have a large number of parameters. A quantization process can be performed on the parameters in an AI model. For example, parameters in an AI model can be quantized from high precision (e.g., 16-bit floating-point numbers (BF16) and 64-bit floating-point numbers (FP64)) to low precision (e.g., 8-bit floating-point numbers (FP8) and 4-bit integer numbers (INT4)), but this disclosure is not limited thereto. In this way, the size of the AI ​​model can be reduced, memory footprint decreased, inference speed increased, and power consumption reduced, enabling it to run on resource-constrained edge devices.

[0043] Artificial intelligence (AI) models may contain numerous outliers in their parameters. Direct quantization of these parameters can lead to significant precision loss due to outliers crowding out the limited precision space after quantization. A better approach is to first perform a mathematical rotation on the parameters to make their distribution more isotropic, thereby improving the fidelity of subsequent quantization operations. This rotation operation requires a meticulously constructed rotation matrix. For example, obtaining the rotation matrix involves complex and computationally intensive search and learning processes. Furthermore, to ensure the mathematical equivalence of the AI ​​model after rotation and quantization, the activations of the AI ​​model also need to be rotated and quantized. In other words, during AI model runtime, the activations must be rotated before quantization to guarantee mathematical equivalence, resulting in substantial computational overhead.

[0044] Furthermore, the aforementioned rotation and quantization operations are suitable for AI models with relatively few weight matrices and highly regular structures, such as dense large language models (DLMs). Highly heterogeneous AI models, such as Mixture-of-Experts (MoE) architectures, introduce dynamic routing networks, shared expert networks, and hundreds or thousands of independent expert sub-networks. For example, a MoE architecture can decompose a large task into multiple smaller tasks, and the router network dynamically selects suitable expert networks to process the smaller tasks. In this way, a large number of parameters can be achieved with low computational cost. However, when the aforementioned rotation and quantization operations are directly applied to these highly heterogeneous AI models, it leads to the problem of mathematical inequivalence between the AI ​​models before and after quantization.

[0045] Figure 1 A flowchart illustrating a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown. Figure 1 As shown, method 1000 includes steps S1100-S1400.

[0046] In step S1100, a rotation matrix for performing rotation operations can be obtained, including left rotation operations and right rotation operations.

[0047] According to at least one embodiment of this disclosure, a rotation matrix can be generated with low computational complexity. The rotation matrix can be used to perform different rotation operations on the parameters of an artificial intelligence model, such as left rotation and right rotation operations.

[0048] In step S1200, the weight matrix of multiple layers of the artificial intelligence model can be obtained.

[0049] According to at least one embodiment of this disclosure, the weight matrix of multiple layers can be the core computational structure of an artificial intelligence model. The weight matrix of each layer can perform a linear transformation on the input data and progressively extract and combine features through matrix multiplication. For example, the weight matrix can capture the basic patterns of the input data (e.g., edges and textures in image recognition, but not limited to these), and integrate these basic patterns into more complex abstract representations (e.g., object parts or the whole in image recognition). The weight matrix of multiple layers can have a huge number of parameters. Therefore, the weight matrix of multiple layers can occupy a large amount of storage space and transmission bandwidth, and will consume a lot of computational resources during operation.

[0050] In step S1300, depending on the type of multiple layers, a rotation matrix can be used to perform either a left rotation operation or a right rotation operation on the weight matrix of each of the multiple layers.

[0051] According to at least one embodiment of this disclosure, the artificial intelligence model can be a highly heterogeneous AI model, such as a large MoE architecture model. Such an AI model can have multiple layers of different types. Different rotation operations, such as left rotation or right rotation, can be applied to different layers depending on their type. In this way, the impact of rotation operations on the mathematical equivalence of the AI ​​model can be offset, thereby ensuring that the reasoning behavior of the AI ​​model after the rotation operation is performed remains unchanged or not significantly altered. Mathematical equivalence can refer to the fact that the input-output mapping relationship of the AI ​​model after transformation (e.g., rotation operation and quantization operation) remains consistent with that of the initial AI model. Furthermore, the weight matrix of the AI ​​model after the rotation operation is closer to isotropy, which will avoid or mitigate severe accuracy loss during quantization, thereby providing feasibility for subsequent quantization operations.

[0052] Furthermore, since the method according to at least one embodiment of this disclosure offsets the impact of rotation operations on the mathematical equivalence of the artificial intelligence model, rotation of the activation is not required during actual use of the artificial intelligence model. In this way, computational resources can be further saved, making it more friendly to edge devices with limited computing resources.

[0053] In step S1400, quantization can be performed on the weight matrices of the multiple layers after the rotation operation.

[0054] Quantization operations can be used to reduce the precision of the weight matrices of multiple layers after a rotation operation. Quantization operations can include symmetric quantization operations, asymmetric quantization operations, etc., but this disclosure is not limited to these.

[0055] According to at least one embodiment of this disclosure, quantization operations can reduce the memory footprint and bandwidth requirements of the weight matrix of an artificial intelligence model, and improve the running speed of the model. However, such quantization operations are contingent upon not significantly sacrificing accuracy and ensuring that the artificial intelligence model is mathematically equivalent before and after performing rotation-based quantization operations; otherwise, the quantization operations would be meaningless.

[0056] The rotation operation performed in step S1300 can make the weight matrix more isotropic while ensuring mathematical equivalence, thereby avoiding significant accuracy loss caused by quantization. In other words, according to at least one embodiment of this disclosure, the memory usage of the weight matrix of multiple layers after performing the above-described rotation-based quantization operation is lower than the memory usage of the weight matrix of multiple layers before performing the quantization operation.

[0057] According to at least one embodiment of this disclosure, the transmission bandwidth of the weight matrix of the multiple layers after performing the above-described rotation-based quantization operation is lower than the transmission bandwidth of the weight matrix of the multiple layers before performing the quantization operation.

[0058] According to at least one embodiment of this disclosure, the running speed of an artificial intelligence model having a weight matrix of multiple layers after performing the above-described rotation-based quantization operation is higher than the running speed of an artificial intelligence model having a weight matrix of multiple layers before performing the quantization operation.

[0059] The artificial intelligence models described above according to at least one embodiment of this disclosure may include large language models, computer vision models, multimodal models, and speech processing models, but this disclosure is not limited thereto. Large language models can process and understand human language, enabling translation, question answering, and generation. The input of a large language model may include text instructions, and the output may include text content, such as answers. Computer vision models can analyze and understand visual information for recognition, detection, and classification. The input of a large language model may include digital images, and the output may include output labels, bounding boxes, or segmentation masks. Multimodal models can comprehensively process various information such as text and images, performing cross-modal reasoning and generation. The input of a multimodal model may include a combination of text and images, and the output may be content-based text answers or new images. Speech processing models can realize the mutual conversion between speech and text, used for dictation and speech synthesis. The input of a speech processing model may include audio or text, and the output may include corresponding transcribed text or synthesized speech.

[0060] Figure 2 A schematic diagram of a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown. Figure 2 Zhongyu Figure 1 Identical or similar parts will not be described again to avoid redundancy.

[0061] According to at least one embodiment of this disclosure, an initial artificial intelligence model 2110 and an initial manifest file 2120 can be obtained. The initial artificial intelligence model 2110 can be a model to be subjected to rotation and quantization operations, and the initial manifest file 2120 can define the plugins and dataset calling specifications of the initial artificial intelligence model 2110.

[0062] According to at least one embodiment of this disclosure, the obtained initial artificial intelligence model 2110 can be divided into multiple slices by the scheduler 2200, and the initial manifest file 2120 can be divided into multiple parts. The scheduler 2200 can further schedule the multiple slices of the initial artificial intelligence model 2110 and the multiple parts of the initial manifest file 2120 to the executors 2310, 2320, ..., 23N0 included in the distributed executor 2300, so as to perform rotation and quantization operations on the weight matrices in the multiple slices in a distributed manner. Each of the executors 2310, 2320, ..., 23N0 included in the distributed executor 2300 can correspond to a computing node. Therefore, multiple computing nodes corresponding to the executors 2310, 2320, ..., 23N0 can be used to perform rotation and quantization operations on the weight matrices of multiple layers in the multiple slices respectively.

[0063] According to at least one embodiment of this disclosure, multiple computing nodes can use the same rotation matrix. For example, according to at least one embodiment of this disclosure, the rotation matrix may include a first rotation matrix and a second rotation matrix. Multiple computing nodes corresponding to executors 2310, 2320, ..., 23N0 can use the same first rotation matrix and second rotation matrix. According to at least one embodiment of this disclosure, a third rotation matrix, as a random Hadamard matrix, can be generated by a first process (e.g., a Rank 0 process) running on a first computing node (e.g., executor 2310) among the multiple computing nodes. For example, the Rank 0 process can use a fixed random neutron to generate a random Hadamard matrix R13 of HIDDEN_SIZE × HIDDEN_SIZE (HIDDEN_SIZE being the hidden layer size of the artificial intelligence model) as the third rotation matrix. A Hadamard matrix is ​​an orthogonal square matrix consisting of +1 and -1 elements. Any two rows of a Hadamard matrix are orthogonal, and calculating the product of a Hadamard matrix and a vector is faster than general matrix multiplication because it can be implemented using the Fast Walsh Transform.

[0064] In other words, the method according to at least one embodiment of this disclosure can use a random Hadamard matrix generated by a fixed random seed as a rotation matrix without going through a complex and computationally intensive search and training process. Therefore, this method can significantly reduce deployment costs and computational overhead while ensuring performance.

[0065] According to at least one embodiment of this disclosure, the Rank 0 process can further calculate the rotation scaling factor of the third rotation matrix R13 and the fourth rotation matrix R14, which is the inverse of the third rotation matrix R13. For example, the Rank 0 process can calculate the rotation scaling factor according to the following equation (1):

[0066] R13_scale = sqrt(HIDDEN_SIZE)(1)

[0067] Where R13_scale is the rotation scaling factor of the third rotation matrix R13, sqrt() is the square root operation, and HIDDEN_SIZE is the hidden layer size of the artificial intelligence model.

[0068] For example, the Rank 0 process can calculate the fourth rotation matrix R14 according to the following equation (2):

[0069] R14 = R13.T(2)

[0070] Here, R14 is the fourth rotation matrix, which is the inverse of the third rotation matrix R13, and .T is the transpose operator.

[0071] According to at least one embodiment of this disclosure, the Rank 0 process can divide the third rotation matrix R13 by the rotation scaling factor R13_scale to obtain the first rotation matrix R11, and divide the fourth rotation matrix R14 by the rotation scaling factor R13_scale to obtain the second rotation matrix R12. In this way, the normalization requirements of high-dimensional Hadamard matrices can be taken into account, thereby avoiding mathematical equivalence in the weight matrix of the artificial intelligence model caused by subsequent rotation operations. For example, the Hadamard matrix H can satisfy H×HT=nI, where n is the dimension of the Hadamard matrix. However, when the Hadamard matrix H and the Hadamard matrix HT are not unit orthogonal, failing to divide by n will introduce a scale shift in subsequent rotation operations, disrupting the mathematical equivalence of the artificial intelligence model before and after performing rotation-based quantization operations.

[0072] According to a method of at least one embodiment of the present disclosure, by dividing the third rotation matrix and the fourth rotation matrix by a rotation scaling factor to obtain the first rotation matrix and the second rotation matrix, the scale shift caused by the rotation matrix to the artificial intelligence model can be eliminated.

[0073] According to at least one embodiment of this disclosure, the Rank 0 process can transmit at least one of a first rotation matrix and a second rotation matrix to remaining processes running on remaining computing nodes (e.g., executors 2320 to executors 23N0) other than the first computing node (e.g., executor 2310), but this disclosure is not limited thereto. For example, the Rank 0 process can transmit one or more of a first rotation matrix, a second rotation matrix, a third rotation matrix, a fourth rotation matrix, and a rotation scaling factor to remaining processes running on remaining computing nodes (e.g., executors 2320 to executors 23N0) other than the first computing node (e.g., executor 2310). The Rank 0 process can synchronize at least one of the first rotation matrix and the second rotation matrix to the remaining processes running on the remaining computing nodes in a distributed communication manner (e.g., broadcast). For example, the Rank 0 process running on executor 2310 can transmit at least one of the aforementioned rotation matrix and rotation scaling factor to scheduler 2200, and scheduler 2200 can transmit at least one of the aforementioned rotation matrix and rotation scaling factor to the remaining executors. Although Figure 2 The scheduler 2200 and executor 2310 are shown as separate entities, but the scheduler 2200 can be merged with the executor 2310. That is, the executor 2310 running the Rank 0 process can act as the scheduler 2200 to directly transfer at least one of the aforementioned rotation matrix and rotation scaling factor to the remaining executors. In this way, it can be ensured that all computing nodes use the same rotation matrix, thereby reducing the computational complexity of obtaining the rotation matrix and maintaining mathematical equivalence.

[0074] Each process running on the multiple computing nodes corresponding to executors 2310, 2320, ..., 23N0 can perform dequantization, rotation, and quantization operations on the scheduled model slices. This reduces the memory footprint and bandwidth consumption of the AI ​​model while maintaining mathematical equivalence, and improves the model's running speed. The aforementioned dequantization, rotation, and quantization operations will refer to... Figure 3 Further description. In addition, each process running on the multiple computing nodes corresponding to executors 2310, 2320, ..., 23N0 can serialize the slices that have undergone rotation and quantization operations, thereby saving the slices for further deployment.

[0075] According to at least one embodiment of this disclosure, executors 2310, 2320, ..., 23N0 can respectively generate model fragments 2410, 2420, ..., 24N0 after performing rotation and quantization operations, and corresponding Manifest files 2510, 2520, ..., 25N0. Aggregator 2600 can aggregate multiple model fragments 2410, 2420, ..., 24N0 obtained using multiple computing nodes after performing rotation and quantization operations into an artificial intelligence model after performing rotation and quantization operations, for example, a merged artificial intelligence model 2710 and a corresponding Manifest file 2720, but this disclosure is not limited thereto. For example, multiple model fragments 2410, 2420, ..., 24N0 may not be aggregated into a merged artificial intelligence model 2710, and the corresponding model fragments may be deployed directly.

[0076] According to at least one embodiment of this disclosure, the aforementioned distributed processing can efficiently handle ultra-large models with hundreds of billions of parameters. Furthermore, layer-by-layer processing and memory cleanup can be implemented through code, thereby optimizing memory usage. By using the generated random Hadamard matrix as the rotation matrix and employing the same rotation matrix across multiple distributed computing nodes, computational efficiency can be improved and engineering implementation complexity reduced. In this way, large-scale rotation matrices can be generated and applied efficiently and consistently in a distributed training / inference environment, thus providing a systematic engineering solution.

[0077] Figure 3 A schematic diagram of a rotation-based quantization method for an artificial intelligence model according to at least one embodiment of the present disclosure is shown. Although Figure 3 The process of performing rotation and quantization operations on a large MoE architecture model is illustrated, but those skilled in the art will understand that the method according to at least one embodiment of this disclosure can be applied to other types of artificial intelligence models. Figure 3 Solid arrows in the diagram represent data streams that have been subjected to a right rotation operation, while dashed arrows represent data streams that have not been subjected to a rotation operation. Figure 3 Zhongyu Figure 1 and Figure 2 Identical or similar parts will not be described again to avoid redundancy.

[0078] like Figure 3 As shown, the MoE architecture large model 3000 may include an embedding layer 3100, root mean square normalization (RMSNorm) layers 3210 and 3220, a dense part 3300, a MoE part 3400, a model normalization (Norm) layer 3500, and an output header layer 3600.

[0079] According to at least one embodiment of this disclosure, in order to avoid the accumulation of large numerical errors during rotation (e.g., particularly significant in hidden layers with large dimensions such as 7168). Figure 3 The rotation operation needs to be performed with a high-precision weight matrix. For example, according to at least one embodiment of this disclosure, the weight matrices of multiple layers of the AI ​​model to be rotated can be at least a predetermined precision. The predetermined precision can be set according to deployment requirements or user needs, for example, FP64. However, some AI models have a precision lower than this predetermined precision when they are released or published. When rotating and quantizing such an AI model, it is necessary to first perform dequantization. For example, a first precision weight matrix of multiple layers of the AI ​​model with a precision lower than the predetermined precision and a corresponding quantization scaling factor can be obtained. The first precision can be FP8, but this disclosure is not limited to this. Dequantization can be performed on the first precision weight matrix at least based on the corresponding quantization scaling factor to obtain a weight matrix with a precision at least a predetermined precision. For example, the precision of the weight matrix with a precision at least a predetermined precision can be BF16, but this disclosure is not limited to this. A rotation operation can be performed on the weight matrix with a precision at least a predetermined precision after dequantization, and then a quantization operation can be performed to obtain a weight matrix with a second precision. The second precision, for example, INT4, can be lower than the first precision of the weight matrix of the initial AI model, for example, FP8. In this way, the parameters of an artificial intelligence model can be quantified without introducing a large accumulation of numerical errors.

[0080] According to at least one embodiment of this disclosure, an artificial intelligence model may include an embedding layer 3100. Taking natural language processing as an example, the embedding layer 3100 can convert discrete symbols into dense vectors, preserve relations through dimensionality reduction, and capture semantic and grammatical patterns to calculate similarity. If the aforementioned rotation matrix is ​​used to directly perform a rotation operation on the weight matrix in the embedding layer 3100, a bias will be introduced, thereby destroying the mathematical equivalence of the artificial intelligence model.

[0081] The method according to at least one embodiment of the present disclosure can perform a centering operation on the weights in the weight matrix of the embedding layer 3100 to eliminate the bias effect generated by the embedding layer 3100.

[0082] According to at least one embodiment of this disclosure, performing a centering operation on the weights in the weight matrix of the embedding layer 3100 may include determining the mean of the weights in each row of the weight matrix of the embedding layer 3100. Further, the centering operation may be performed by subtracting the mean of the row in which each weight is located from the weight matrix of the embedding layer 3100. In this way, the weights in the weight matrix of the embedding layer 3100 can be distributed with 0 as the center.

[0083] According to at least one embodiment of this disclosure, a rotation matrix can be used to perform a right rotation operation on the weight matrix of the embedded layer 3100, which is subjected to a centering operation. As described above, the rotation matrix may include a first rotation matrix R11 and a second rotation matrix R12, which is the inverse of the first rotation matrix. Hereinafter, performing a left rotation operation on the weight matrix of a portion of the layer may refer to left-multiplying the weight matrix using the second rotation matrix R12. The specific process of the left rotation operation can be shown in the following equation (3):

[0084] R12 @ W(3)

[0085] Where R12 is the second rotation matrix, W is the weight matrix, and @ is the matrix multiplication operator.

[0086] In the following text, performing a right rotation on the weight matrix of another layer can refer to multiplying the weight matrix on the right using the first rotation matrix R11. The specific process of the right rotation operation can be shown in the following equation (4):

[0087] W @ R11 (4)

[0088] Where R11 is the first rotation matrix, W is the weight matrix, and @ is the matrix multiplication operator.

[0089] According to at least one embodiment of this disclosure, the weight matrices of RMSNorm layers 3210 and 3220 can be diagonally scaled matrices. The root mean square normalization layer can use root mean square values ​​to scale the activations, thereby stabilizing network training and improving computational efficiency. When directly performing rotation operations on RMSNorm layers 3210 and 3220, the invariance of the rotation process cannot be satisfied. RMSNorm layer 3210 can be an input root mean square normalization layer, and RMSNorm layer 3220 can be a post-attention root mean square normalization layer. To eliminate the mathematical equivalence effect of RMSOrm layers 3210 and 3220 on rotation operations, the weight matrices of RMSOrm layers 3210 and 3220 can be merged into the weight matrix of the linear projection layer immediately following the root mean square normalization layer, and the weights in the weight matrix of the root mean square normalization layer (e.g., RMSOrm layers 3210 and 3220) can be set to one. Figure 3 As shown, the linear projection layer immediately following the RMSNorm layer 3210 may include a query Q projection layer 3310 and a key K / value V projection layer 3320. The linear projection layer immediately following the RMSNorm layer 3220 may include at least one of an up-dimensional projection layer, a gate projection layer, and a router projection layer.

[0090] According to at least one embodiment of this disclosure, the weight matrix of the linear projection layer immediately following the root mean square normalization layer can be left-multiplied by the weight matrix of the root mean square normalization layer to obtain the fused weight matrix of the linear projection layer immediately following the root mean square normalization layer. Specifically, the fusion process can be represented by the following equation (5):

[0091] W_linear_fused = W_linear @ diag(w_norm) (5)

[0092] Where W_linear_fused is the weight matrix after the fusion operation of the linear projection layer, and W_linear is the weight matrix before the fusion operation of the linear projection layer, diag(w_norm) is the weight matrix of the root mean square normalization layer, and @ is the matrix multiplication operator.

[0093] It is important to note that the fusion operation, which merges the weight matrix of the root mean square normalized layer into the weight matrix of the immediately following linear projection layer, needs to be performed before applying either a left or right rotation to the weight matrices of the multiple layers (e.g., the linear projection layer) using rotation matrices. Otherwise, mathematical inequivalence issues arise due to the non-commutative nature of scaling and rotation. Therefore, applying either a left or right rotation to the weight matrices of the multiple layers using rotation matrices includes performing at least one of a left or right rotation on the weight matrix of the immediately following linear projection layer after the fusion operation.

[0094] like Figure 3 As shown, the MoE architecture large model 3000 can include a dense part 3300 and a MoE part 3400. The dense part 3300 can globally model the sequence context relationship through a self-attention mechanism, integrate information, and provide a high-quality input representation for the subsequent MoE part 3400.

[0095] The MoE part 3400 can consist of numerous expert networks, such as shared expert 3420, independent experts 3430, 3440...34N0. By selectively activating the expert networks through the router projection layer 3410, the MoE part 3400 can significantly increase model capacity without significantly increasing computational overhead.

[0096] like Figure 3As shown, the dense portion 3300 may include a Q-projection layer 3310, a K / V-projection layer 3320, and an O-projection layer 3330. The Q-projection layer 3310 can map the input to a query vector to actively retrieve information. The K / V-projection layer 3320 can map the input to key-value pairs to store the content to be searched. The O-projection layer 3330 can fuse attention results to output a new context vector. According to at least one embodiment of this disclosure, a rotation matrix can be used to perform a left rotation operation on the weight matrix (e.g., after a fusion operation) of at least one of the Q-projection layer 3310 and the K / V-projection layer 3320 among the multiple layers, and a rotation matrix can be used to perform a right rotation operation on the weight matrix of the O-projection layer 3330 among the multiple layers.

[0097] like Figure 3 As shown, the MoE part 3400 may include a router projection layer 3410, up projection layers 3421 and 3431, gate projection layers 3422 and 3432, and down projection layers 3423 and 3433. Up projection layers 3421 and 3431 can increase the feature dimension, thereby enhancing the model's expressive power. Gate projection layers 3422 and 3432 can generate gating signals to regulate the direction of information flow. Down projection layers 3423 and 3433 can compress the feature dimension, thereby integrating information and outputting it. According to at least one embodiment of this disclosure, a rotation matrix can be used to perform a left rotation operation on the weight matrix (e.g., after performing a fusion operation) of at least one of the Router projection layer 3410, Up projection layers 3421 and 3431, and Gate projection layers 3422 and 3432, and a right rotation operation can be performed on the weight matrix of at least one of the Down projection layers 3423 and 3433, and a right rotation operation can be performed on the weight matrix of at least one of the Down projection layers 3423 and 3433. In this way, the impact of rotation operations on the mathematical equivalence of artificial intelligence models can be offset.

[0098] According to at least one embodiment of this disclosure, the model normalization layer 3500 can stabilize the final representation distribution, providing normalized input to the output head layer 3600. Taking natural language processing as an example, the output head layer 3600 can map the hidden state to word probabilities, thereby generating prediction results. Figure 3 As shown, the output head layer 3600 can be set immediately after the model normalization layer 3500. According to at least one embodiment of this disclosure, a rotation matrix can be used to perform a left rotation operation on the weight matrix of the output head layer 3600.

[0099] According to at least one embodiment of this disclosure, the weight matrix of the model normalization layer 3500 can be a diagonally scaled matrix. When a rotation operation is directly performed on the model normalization layer 3500, the invariance of the rotation process cannot be satisfied. To eliminate the influence of the mathematical equivalence of the model normalization layer 3500 on the rotation operation, the weight matrix of the model normalization layer 3500 can be merged into the weight matrix of the output head layer immediately adjacent to the model normalization layer 3500, and the weights in the weight matrix of the model normalization layer are set to one. According to at least one embodiment of this disclosure, the weight matrix of the output head layer 3600, which is set immediately after the model normalization layer 3500, can be left-multiplied by the weight matrix of the model normalization layer 3500 to obtain the merged weight matrix of the output head layer 3600.

[0100] It is important to note that the fusion operation, which merges the weight matrix of the normalized layer 3500 into the weight matrix of the immediately following output head layer 3600, must be performed before rotating the weight matrix of the output head layer 3600 using a rotation matrix. Otherwise, mathematical inequivalence will occur due to the non-commutative nature of scaling and rotation. Therefore, performing a left rotation on the weight matrix of the output head layer 3600 using a rotation matrix involves performing a left rotation on the weight matrix of the output head layer 3600 after the fusion operation.

[0101] According to at least one embodiment of this disclosure, quantization operations can be further performed on the layers in the MoE architecture large model 3000 that have undergone the above-mentioned rotation operations, so as to reduce the memory usage and transmission bandwidth of the weight matrix and improve the running speed of the artificial intelligence model.

[0102] According to at least one embodiment of the method disclosed herein, in the rotation-based quantization process for large language models (e.g., MoE architecture large models) applied to natural language processing and reasoning, the accuracy of the artificial intelligence model can be maintained under low-bit (e.g., FP8, INT4) quantization conditions, while ensuring mathematical equivalence and deployment compatibility. The method according to at least one embodiment of the method disclosed herein fully considers the complex and highly heterogeneous structure in the MoE architecture large model, and designs detailed and unified rotation operations for the router network, shared expert network, and numerous independent expert networks in the MoE architecture large model, thereby ensuring mathematical equivalence.

[0103] Furthermore, the method according to at least one embodiment of this disclosure performs only the aforementioned simplified rotation operation on each layer, without performing further rotation operations on specific layers (e.g., Q-layer, K / V-layer, O-layer, and Down-layer). In this way, by reducing the number of rotation operations with low returns based on the fact that the weight matrices of each layer tend to be isotropic, computational overhead can be reduced. This is especially important when the hidden layer dimension is large and the number of layers is extremely deep.

[0104] The method according to at least one embodiment of this disclosure achieves globally mathematically equivalent low-bit quantization in the MoE architecture by performing rotation, centering, and fusion operations on the layers of the artificial intelligence model, thereby systematically solving the application problem of rotation-based quantization operations on complex MoE models. Furthermore, by performing a fusion operation on the weight matrices of the root mean square normalized layer and the model normalized layer, modification of the inference code can be avoided, thus reducing the complexity of use.

[0105] Figure 4 A schematic structural diagram of an artificial intelligence processor according to at least one embodiment of the present disclosure is shown.

[0106] Artificial intelligence processors may include graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), tensor processing units (TPUs), deep learning processing units (DPUs), accelerated processing units (APUs), neural network processing units (NPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs), but this disclosure is not limited thereto.

[0107] like Figure 4 As shown, an example of an artificial intelligence processor is an array of programmable multiprocessors that can be used in the quantization methods of this disclosure. For example, the programmable multiprocessors of the artificial intelligence processor can be a Streaming Processor Cluster (SPC), such as including streaming processor cluster 1, ..., streaming processor cluster M as shown in the figure, where M is a positive integer greater than 1. In the artificial intelligence processor, one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0108] like Figure 4 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 4The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple Cores (also called computational cores), each of which includes an Arithmetic Logic Unit (ALU), a floating-point unit, etc. These Cores are used to execute specific computational tasks. Furthermore, a Computation Unit may also include registers (e.g., ...). Figure 4 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0109] like Figure 4 As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0110] like Figure 4 As shown, each streaming processor cluster also provides a buffer for caching data from the N computing units within the streaming processor cluster.

[0111] In parallel computing, computational tasks are typically executed through multiple threads. These threads are divided into multiple thread blocks before execution in the artificial intelligence processor (or general-purpose graphics processor or parallel computing processor), and then dispatched via a thread block dispatch module. Figure 4 (Not shown in the diagram) Multiple thread blocks are distributed to various computing units. All threads in a thread block must be assigned to the same computing unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple threads from multiple thread blocks can execute on multiple computing cores within the same computing unit, or on multiple computing cores in different computing units, sharing registers, shared memory, and other data transfer mechanisms.

[0112] In each computing unit, the thread beam scheduling / distribution module ( Figure 4(Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 4 It is used for reading and writing operations in high-bandwidth memory (HBM).

[0113] Figure 4 The AI ​​processor shown can execute the rotation-based quantization method for AI models described above, including: obtaining a rotation matrix for performing rotation operations, the rotation operations including left rotation and right rotation; obtaining weight matrices for multiple layers of the AI ​​model; performing one of the left rotation and right rotation operations on the weight matrices of the multiple layers respectively using the rotation matrix, depending on the type of the multiple layers; and performing a quantization operation on the weight matrices of the multiple layers after performing the rotation operations.

[0114] For example, the method according to at least one embodiment of the present disclosure further includes: dividing the obtained artificial intelligence model into multiple slices; using multiple computing nodes to perform rotation and quantization operations on the weight matrices of multiple layers of the divided multiple slices, respectively, wherein the multiple computing nodes use the same rotation matrix; and aggregating the multiple slices obtained by performing rotation and quantization operations using the multiple computing nodes into the artificial intelligence model after performing rotation and quantization operations.

[0115] For example, according to a method of at least one embodiment of the present disclosure, wherein the rotation matrix includes a first rotation matrix and a second rotation matrix, the method further includes: generating a third rotation matrix as a random Hadamard matrix by a first process running on a first computing node among a plurality of computing nodes; calculating a rotation scaling factor of the third rotation matrix and a fourth rotation matrix as the inverse of the third rotation matrix by the first process; dividing the third rotation matrix by the rotation scaling factor to obtain the first rotation matrix, and dividing the fourth rotation matrix by the rotation scaling factor to obtain the second rotation matrix; and transmitting one or more of the first rotation matrix, the second rotation matrix, the third rotation matrix, the fourth rotation matrix, and the rotation scaling factor by the first process to remaining processes running on remaining computing nodes other than the first computing node among the plurality of computing nodes.

[0116] For example, according to at least one embodiment of the method of this disclosure, one of performing a left rotation operation and a right rotation operation on the weight matrices of the multiple layers respectively using a rotation matrix, depending on the type of the multiple layers, further includes: performing a left rotation operation on the weight matrix of at least one of the multiple layers, namely a query Q projection layer, a key K projection layer, a value V projection layer, an up-dimensional projection layer, and a gated projection layer; and performing a right rotation operation on the weight matrix of at least one of the multiple layers, namely an output O projection layer and a down-dimensional projection layer.

[0117] For example, according to at least one embodiment of the method of this disclosure, the rotation matrix includes a first rotation matrix and a second rotation matrix that is the inverse of the first rotation matrix, and one of performing a left rotation operation and a right rotation operation on the weight matrices of multiple layers using the rotation matrix further includes: multiplying the weight matrix of a first portion of the multiple layers by the second rotation matrix to the left to perform a left rotation operation; and multiplying the weight matrix of a second portion of the multiple layers by the first rotation matrix to the right to perform a right rotation operation.

[0118] For example, according to at least one embodiment of the method of this disclosure, the plurality of layers include a root mean square normalization layer and a linear projection layer disposed immediately after the root mean square normalization layer. The method further includes: fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer; and setting the weights in the weight matrix of the root mean square normalization layer to one.

[0119] For example, according to at least one embodiment of the present disclosure, the method of fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer immediately following the root mean square normalization layer includes: left-multiplying the weight matrix of the linear projection layer immediately following the root mean square normalization layer by the weight matrix of the root mean square normalization layer to obtain the fused weight matrix of the linear projection layer immediately following the root mean square normalization layer.

[0120] For example, according to a method of at least one embodiment of the present disclosure, the root mean square normalization layer includes at least one of an input root mean square normalization layer and a post-attention root mean square normalization layer; and the linear projection layer immediately following the root mean square normalization layer includes at least one of a query Q projection layer, a key K projection layer, a value V projection layer, an up-dimensional projection layer, a gate projection layer, and a router projection layer.

[0121] For example, according to at least one embodiment of the method of this disclosure, the fusion of the weight matrix of the root mean square normalized layer into the weight matrix of the linear projection layer immediately following the root mean square normalized layer is performed before performing one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using a rotation matrix; and wherein performing one of the left rotation operation and right rotation operation on the weight matrices of the plurality of layers using a rotation matrix includes performing at least one of a left rotation operation and a right rotation operation on the weight matrix of the linear projection layer immediately following the root mean square normalized layer after the fusion operation using a rotation matrix.

[0122] For example, according to a method of at least one embodiment of the present disclosure, wherein a plurality of layers include an embedded layer, and the method further includes: performing a centering operation on the weights in the weight matrix of the embedded layer; and performing a right rotation operation on the weight matrix of the embedded layer, which has undergone the centering operation, using a rotation matrix.

[0123] For example, according to a method of at least one embodiment of the present disclosure, performing a centering operation on the weights in the weight matrix of the embedded layer includes: determining the mean of the weights in each row of the weight matrix of the embedded layer; and subtracting the mean of the row in which the weight is located from each weight in the weight matrix of the embedded layer to perform a centering operation.

[0124] For example, according to at least one embodiment of the method of this disclosure, wherein the plurality of layers include an output head layer, and one of performing a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers respectively using a rotation matrix further includes: performing a left rotation operation on the weight matrix of the output head layer using a rotation matrix.

[0125] For example, according to at least one embodiment of the method of this disclosure, wherein a plurality of layers include a model normalization layer and an output head layer is set immediately after the model normalization layer, the method further includes: fusing the weight matrix of the model normalization layer into the weight matrix of the output head layer set immediately after the model normalization layer; and setting the weights in the weight matrix of the model normalization layer to one, wherein performing a left rotation operation on the weight matrix of the output head layer using a rotation matrix includes: performing a left rotation operation on the weight matrix of the output head layer after the fusion operation using a rotation matrix.

[0126] For example, according to a method of at least one embodiment of the present disclosure, obtaining a weight matrix of multiple layers of an artificial intelligence model includes: obtaining a weight matrix of multiple layers of the artificial intelligence model with a precision not lower than a predetermined value.

[0127] For example, according to at least one embodiment of the method of this disclosure, obtaining a weight matrix of multiple layers of an artificial intelligence model with a precision not lower than a predetermined precision includes: obtaining a weight matrix of multiple layers of the artificial intelligence model with a first precision and a corresponding quantization scaling factor, wherein the first precision is lower than the predetermined precision; performing dequantization on the weight matrix of the first precision at least based on the corresponding quantization scaling factor to obtain a weight matrix with a precision not lower than the predetermined precision.

[0128] For example, according to at least one embodiment of the method of this disclosure, performing a quantization operation on the weight matrix of multiple layers after performing a rotation operation includes: performing a quantization operation on the weight matrix of multiple layers after performing a rotation operation to obtain a weight matrix with a second precision, wherein the second precision is lower than the first precision.

[0129] For example, according to a method of at least one embodiment of the present disclosure, the artificial intelligence model includes a hybrid expert model.

[0130] For example, according to at least one embodiment of the method of this disclosure, the weight matrix of the multiple layers after performing the quantization operation satisfies one or more of the following: the memory usage of the weight matrix of the multiple layers after performing the quantization operation is lower than the memory usage of the weight matrix of the multiple layers before performing the quantization operation; the transmission bandwidth of the weight matrix of the multiple layers after performing the quantization operation is lower than the transmission bandwidth of the weight matrix of the multiple layers before performing the quantization operation; and the running speed of the artificial intelligence model having the weight matrix of the multiple layers after performing the quantization operation is higher than the running speed of the artificial intelligence model having the weight matrix of the multiple layers before performing the quantization operation.

[0131] Figure 5 A schematic diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0132] like Figure 5 As shown, the electronic device 5000 may include at least one processing unit 5100 and at least one memory 5200. The at least one memory 5200 includes one or more computer program modules 5210. The one or more computer program modules 5210 are stored in the at least one memory 5200 and can be configured to be read and executed by the at least one processing unit 5100. The one or more computer program modules 5210 include instructions for performing the various methods described above according to at least one embodiment of the present disclosure. When executed by the at least one processing unit 5100, they can perform one or more steps of the various methods described above according to at least one embodiment of the present disclosure and their additional aspects.

[0133] At least one memory 5200 and at least one processing unit 5100 may be interconnected via a bus system and / or other forms of connection mechanism (not shown). For example, the bus may be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0134] Exemplarily, at least one processing unit 5100 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or program execution capabilities, such as a field-programmable gate array (FPGA). At least one processing unit 5100 may be a general-purpose processor or a dedicated processor, capable of controlling other components in the electronic device 5000 to perform desired functions. According to at least one embodiment of this disclosure, the electronic device 5000 may be such as... Figure 4 The artificial intelligence processor shown.

[0135] Exemplarily, at least one memory 5200 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules 5210 may be stored on the computer-readable storage medium, and at least one processing unit 5100 may run one or more computer program modules 5210 to implement various functions of the electronic device 5000. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0136] For example, electronic device 5000 may also include input devices such as cameras, touchscreens, touchpads, keyboards, mice, webcams, microphones, accelerometers, and gyroscopes; output devices such as liquid crystal displays, speakers, and vibrators; storage devices such as magnetic tapes and hard disks (HDDs or SDDs); and communication devices such as network interface cards like LAN cards and modems. The communication devices allow electronic device 5000 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. A drive is connected to the I / O interface as needed. Removable storage media, such as disks, optical disks, magneto-optical disks, and semiconductor memories, are installed on the drive as needed so that computer programs read from them can be installed into the storage device as required.

[0137] For example, the electronic device 5000 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device can communicate wirelessly with networks and other devices, such as the Internet, intranets and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0138] The electronic device 5000 may be, for example, a system-on-a-chip (SOC) or a device including the SOC. For instance, it can be any device such as a mobile phone, tablet computer, laptop computer, e-reader, game console, television, digital photo frame, navigator, home appliance, communication base station, industrial controller, server, etc., or any combination of data processing devices and hardware. The embodiments of this disclosure do not limit this. The specific functions and technical effects of the electronic device 5000 can be found in the foregoing description of the method and its additional aspects according to at least one embodiment of this disclosure, and will not be repeated here.

[0139] Figure 6 A schematic diagram of a quantization apparatus according to at least one embodiment of the present disclosure is shown. Figure 6The quantization device 6000 shown can be a rotation-based quantization device for artificial intelligence models. For example... Figure 6 As shown, the quantization device 6000 may include a rotation matrix acquisition module 6100, a weight matrix acquisition module 6200, a rotation module 6300, and a quantization module 6400.

[0140] According to at least one embodiment of the present disclosure, the rotation matrix obtaining module 6100 can be configured to obtain a rotation matrix for performing rotation operations, wherein the rotation operations include left rotation operations and right rotation operations.

[0141] According to at least one embodiment of this disclosure, the weight matrix obtaining module 6200 can be configured to obtain the weight matrix of multiple layers of an artificial intelligence model.

[0142] According to at least one embodiment of the present disclosure, the rotation module 6300 can be configured to perform one of a left rotation operation and a right rotation operation on the weight matrix of the multiple layers respectively, based on the type of the multiple layers.

[0143] According to at least one embodiment of the present disclosure, the quantization module 6400 can be configured to perform a quantization operation on the weight matrix of multiple layers after a rotation operation.

[0144] According to at least one embodiment of this disclosure, the quantization device 6000 may further include a partitioning module, which is configured to: partition the obtained artificial intelligence model into multiple pieces; perform rotation and quantization operations on the weight matrices of multiple layers of the partitioned multiple pieces using multiple computing nodes, wherein the multiple computing nodes use the same rotation matrix; and aggregate the multiple pieces obtained by performing rotation and quantization operations using the multiple computing nodes into an artificial intelligence model after performing rotation and quantization operations.

[0145] According to at least one embodiment of this disclosure, the rotation matrix includes a first rotation matrix and a second rotation matrix, and the rotation module 6300 may further be configured to: generate a third rotation matrix as a random Hadamard matrix by a first process running on a first computing node among a plurality of computing nodes; calculate a rotation scaling factor of the third rotation matrix and a fourth rotation matrix as the inverse of the third rotation matrix by the first process; divide the third rotation matrix by the rotation scaling factor to obtain the first rotation matrix, and divide the fourth rotation matrix by the rotation scaling factor to obtain the second rotation matrix; and transmit one or more of the first rotation matrix, the second rotation matrix, the third rotation matrix, the fourth rotation matrix, and the rotation scaling factor by the first process to the remaining processes running on the remaining computing nodes other than the first computing node among the plurality of computing nodes.

[0146] According to at least one embodiment of this disclosure, the rotation module 6300 may also be configured to perform a left rotation operation on the weight matrix of at least one of the query projection layer, key / value projection layer, dimensionality-up projection layer, gated projection layer, and router projection layer in a plurality of layers using a rotation matrix; and to perform a right rotation operation on the weight matrix of at least one of the output projection layer and dimensionality-down projection layer in a plurality of layers using a rotation matrix.

[0147] According to at least one embodiment of this disclosure, the rotation matrix includes a first rotation matrix and a second rotation matrix that is the inverse of the first rotation matrix, and the rotation module 6300 can also be configured to: perform a left rotation operation by left multiplying the weight matrix of a first portion of a plurality of layers using the second rotation matrix; and perform a right rotation operation by right multiplying the weight matrix of a second portion of a plurality of layers using the first rotation matrix.

[0148] According to at least one embodiment of this disclosure, the plurality of layers include a root mean square normalization layer and a linear projection layer disposed immediately after the root mean square normalization layer, and the quantization device 6000 may further include a fusion module, which may be configured to: fuse the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer; and set the weights in the weight matrix of the root mean square normalization layer to one.

[0149] According to at least one embodiment of this disclosure, the fusion module can also be configured to: multiply the weight matrix of the linear projection layer immediately following the root mean square normalization layer by the weight matrix of the root mean square normalization layer to obtain the fused weight matrix of the linear projection layer immediately following the root mean square normalization layer.

[0150] According to at least one embodiment of this disclosure, the root mean square normalization layer includes at least one of an input root mean square normalization layer and a post-attention root mean square normalization layer; and the linear projection layer immediately following the root mean square normalization layer includes at least one of a query projection layer, a key / value projection layer, an upscaling projection layer, a gated projection layer, and a router projection layer.

[0151] According to at least one embodiment of this disclosure, the fusion module may further be configured to fuse the weight matrix of the root mean square normalized layer into the weight matrix of the linear projection layer immediately following the root mean square normalized layer before performing either a left rotation operation or a right rotation operation on the weight matrices of the multiple layers respectively using a rotation matrix; and wherein the rotation module 6300 may further be configured to perform at least one of a left rotation operation and a right rotation operation on the weight matrix of the linear projection layer immediately following the root mean square normalized layer after the fusion operation using a rotation matrix.

[0152] According to at least one embodiment of the present disclosure, the plurality of layers include an embedded layer, and the rotation module 6300 may also be configured to: perform a centering operation on the weights in the weight matrix of the embedded layer; and perform a right rotation operation on the weight matrix of the embedded layer, which has been centered, using a rotation matrix.

[0153] According to at least one embodiment of this disclosure, the rotation module 6300 may also be configured to: determine the mean of the weights of each row in the weight matrix of the embedding layer; and subtract the mean of the row in which the weight is located from each weight in the weight matrix of the embedding layer to perform a centering operation.

[0154] According to at least one embodiment of the present disclosure, multiple layers include an output head layer, and the rotation module 6300 can also be configured to perform a left rotation operation on the weight matrix of the output head layer using a rotation matrix.

[0155] According to at least one embodiment of this disclosure, multiple layers include a model normalization layer, and the output head layer is set immediately after the model normalization layer. The fusion module can also be configured to: fuse the weight matrix of the model normalization layer into the weight matrix of the output head layer set immediately after the model normalization layer; and set the weights in the weight matrix of the model normalization layer to one. The rotation module 6300 can also be configured to: perform a left rotation operation on the weight matrix of the output head layer after the fusion operation using a rotation matrix.

[0156] According to at least one embodiment of the present disclosure, the weight matrix obtaining module 6200 can also be configured to obtain a weight matrix of multiple layers of an artificial intelligence model with a precision not lower than a predetermined value.

[0157] According to at least one embodiment of this disclosure, the weight matrix obtaining module 6200 may further be configured to: obtain a weight matrix of multiple layers of an artificial intelligence model with a first precision and a corresponding quantization scaling factor, wherein the first precision is lower than a predetermined precision; and perform dequantization on the weight matrix of the first precision at least based on the corresponding quantization scaling factor to obtain a weight matrix with a precision not lower than the predetermined precision.

[0158] According to at least one embodiment of the present disclosure, the quantization module 6400 may also be configured to perform a quantization operation on the weight matrices of multiple layers after a rotation operation to obtain a weight matrix with a second precision, wherein the second precision is lower than the first precision.

[0159] According to at least one embodiment of this disclosure, the artificial intelligence model includes a large model of a hybrid expert architecture.

[0160] According to at least one embodiment of this disclosure, the weight matrix of the multiple layers after performing quantization operation satisfies one or more of the following: the memory usage of the weight matrix of the multiple layers after performing quantization operation is lower than the memory usage of the weight matrix of the multiple layers before performing quantization operation; the transmission bandwidth of the weight matrix of the multiple layers after performing quantization operation is lower than the transmission bandwidth of the weight matrix of the multiple layers before performing quantization operation; and the running speed of the artificial intelligence model having the weight matrix of the multiple layers after performing quantization operation is higher than the running speed of the artificial intelligence model having the weight matrix of the multiple layers before performing quantization operation.

[0161] Figure 7 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0162] like Figure 7 As shown, the electronic device 300 is, for example, suitable for implementing the data transmission method for an artificial intelligence processor provided in embodiments of this disclosure. It should be noted that... Figure 7 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0163] For example, such as Figure 7 As shown, in some examples, electronic device 300 includes a processing device (e.g., including as referenced). Figure 4 The AI ​​processor 301 shown is a parallel processor such as a graphics processor or a general-purpose graphics processor. It can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303 to achieve various functions, such as performing a rotation-based quantization method for an artificial intelligence model provided in at least one embodiment of this disclosure.

[0164] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the rotation-based quantization method for artificial intelligence models according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the rotation-based quantization method for artificial intelligence models can be found in the relevant descriptions in the embodiments of the rotation-based quantization method for artificial intelligence models described above; repeated details will not be repeated here.

[0165] RAM 303 also stores various programs and data required for the operation of the computer system. Processing device 301, ROM 302, and RAM 303 are connected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0166] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0167] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage device 308 as needed. Although Figure 7 An electronic device 300 including various devices is shown; however, it should be understood that implementation or inclusion of all shown devices is not required. More or fewer devices may be implemented or included alternatively. For example, a processor 301 may control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a parallel processor, an artificial intelligence processor, a tensor processor (TPU), or a graphics processing unit (GPU), etc., possessing data processing capabilities and / or program execution capabilities. The CPU may be based on an x86 or ARM architecture, etc. The GPU may be integrated directly onto the motherboard or built into the motherboard's northbridge chip. The GPU may also be built into the CPU.

[0168] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 309 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0169] For example, electronic device 300 can be any device such as mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, navigator, etc., or any combination of electronic devices and hardware. The embodiments disclosed herein do not limit this.

[0170] For example, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, the rotation-based quantization method for artificial intelligence models defined in the embodiments of this disclosure is performed.

[0171] According to at least one embodiment of this disclosure, the electronic device 300 may include the electronic device or quantization device described in the above embodiments.

[0172] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0173] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. For example, a computer-readable storage medium provided according to at least one embodiment of the present disclosure may store instructions thereon that, when executed by one or more processors, cause the processors to perform the methods described above.

[0174] The quantization method, electronic device, storage medium, electronic device, and quantization apparatus provided according to at least one embodiment of the present disclosure can achieve mathematically equivalent quantization operations for highly structurally heterogeneous artificial intelligence models with low computational complexity.

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0177] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0178] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0179] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0180] The following points should be noted regarding this disclosure:

[0181] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0182] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0183] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A rotation-based quantization method for an artificial intelligence model, characterized in that, comprising: obtaining a rotation matrix for performing a rotation operation, wherein the rotation operation comprises a left rotation operation and a right rotation operation; obtaining weight matrices of a plurality of layers of the artificial intelligence model; performing one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix according to types of the plurality of layers, respectively; and performing a quantization operation on the weight matrices of the plurality of layers after performing the rotation operation.

2. The method of claim 1, wherein, further comprising: dividing the obtained artificial intelligence model into a plurality of shards; performing the rotation operation and the quantization operation on the weight matrices of the plurality of layers of the divided plurality of shards using a plurality of computing nodes, respectively, wherein the plurality of computing nodes use the same rotation matrix; and aggregating the plurality of shards obtained using the plurality of computing nodes after performing the rotation operation and the quantization operation into the artificial intelligence model after performing the rotation operation and the quantization operation. the rotation matrix comprises a first rotation matrix and a second rotation matrix, and the method further comprises:

3. The method of claim 2, wherein, generating, by a first process running on a first computing node of a plurality of computing nodes, a third rotation matrix as a random Hadamard matrix; calculating, by the first process, a rotation scaling factor of the third rotation matrix and a fourth rotation matrix as an inverse matrix of the third rotation matrix; dividing the third rotation matrix by the rotation scaling factor to obtain the first rotation matrix and dividing the fourth rotation matrix by the rotation scaling factor to obtain the second rotation matrix; and transmitting, by the first process, one or more of the first rotation matrix, the second rotation matrix, the third rotation matrix, the fourth rotation matrix, and the rotation scaling factor to remaining processes running on remaining computing nodes of the plurality of computing nodes except the first computing node. the performing one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix according to types of the plurality of layers, respectively, further comprises:

4. The method of claim 1, wherein, performing a left rotation operation on a weight matrix of at least one of a query projection layer, a key / value projection layer, a dimension-up projection layer, a gate projection layer, a router projection layer of the plurality of layers using the rotation matrix; and performing a right rotation operation on a weight matrix of at least one of an output projection layer, a dimension-down projection layer of the plurality of layers using the rotation matrix. the rotation matrix comprises a first rotation matrix and a second rotation matrix that is an inverse matrix of the first rotation matrix, and the performing one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers using the rotation matrix, respectively, further comprises:

5. The method of claim 1, wherein, left-multiplying a weight matrix of a first portion of layers of the plurality of layers by the second rotation matrix to perform a left rotation operation; and right-multiplying a weight matrix of a second portion of layers of the plurality of layers by the first rotation matrix to perform a right rotation operation. the plurality of layers comprises a root mean square normalization layer, and a linear projection layer disposed immediately after the root mean square normalization layer, and the method further comprises:

6. The method of claim 1, wherein, fusing a weight matrix of the root mean square normalization layer into a weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer; and ​ set weights in a weight matrix of the root mean square normalization layer to one.

7. The method of claim 6, wherein, the fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer comprises: multiplying the weight matrix of the root mean square normalization layer by a weight matrix of a linear projection layer disposed immediately after the root mean square normalization layer to obtain a fused weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer.

8. The method of claim 6, wherein, the root mean square normalization layer comprises at least one of an input root mean square normalization layer and a post-attention root mean square normalization layer; and the linear projection layer disposed immediately after the root mean square normalization layer comprises at least one of a query projection layer, a key / value projection layer, a dimension-up projection layer, a gate projection layer, and a router projection layer.

9. The method of claim 6, wherein, the fusing the weight matrix of the root mean square normalization layer into the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer is performed before using the rotation matrix to perform one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers; and wherein the using the rotation matrix to perform one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers comprises using the rotation matrix to perform at least one of a left rotation operation and a right rotation operation on the weight matrix of the linear projection layer disposed immediately after the root mean square normalization layer after the performing the fusing operation.

10. The method of claim 1, wherein, the plurality of layers comprises an embedding layer, and the method further comprises: performing a centering operation on weights in a weight matrix of the embedding layer; and using the rotation matrix to perform a right rotation operation on the weight matrix of the embedding layer after the performing the centering operation.

11. The method of claim 10, wherein, the performing the centering operation on weights in a weight matrix of the embedding layer comprises: determining a mean of weights of each row in the weight matrix of the embedding layer; and subtracting the mean of the row in which the weight is located from each weight in the weight matrix of the embedding layer to perform the centering operation.

12. The method of claim 1, wherein, the plurality of layers comprises an output head layer, and the using the rotation matrix to perform one of a left rotation operation and a right rotation operation on the weight matrices of the plurality of layers further comprises: using the rotation matrix to perform a left rotation operation on a weight matrix of the output head layer.

13. The method of claim 12, wherein, the plurality of layers comprises a model normalization layer, and the output head layer is disposed immediately after the model normalization layer, the method further comprising: fusing a weight matrix of the model normalization layer into a weight matrix of the output head layer disposed immediately after the model normalization layer; and setting weights in the weight matrix of the model normalization layer to one, wherein the using the rotation matrix to perform a left rotation operation on the weight matrix of the output head layer comprises using the rotation matrix to perform a left rotation operation on the weight matrix of the output head layer after the performing the fusing operation.

14. The method of any one of claims 1-13, wherein, the obtaining the weight matrices of the plurality of layers of the artificial intelligence model comprises: obtaining weight matrices of a plurality of layers of the artificial intelligence model that are not less than a predetermined precision.

15. The method of claim 14, wherein, the obtaining the weight matrices of the plurality of layers of the artificial intelligence model that are not less than a predetermined precision comprises: obtain a weight matrix of a plurality of layers of the artificial intelligence model at a first precision, wherein the first precision is lower than the predetermined precision, and a corresponding quantization scaling factor; perform dequantization on the weight matrix at the first precision based on at least the corresponding quantization scaling factor to obtain the weight matrix at the no-lower-than-predetermined precision.

16. The method of claim 15, wherein, performing the quantization operation on the weight matrix of the plurality of layers after the rotation operation comprises: performing the quantization operation on the weight matrix of the plurality of layers after the rotation operation to obtain a weight matrix at a second precision, wherein the second precision is lower than the first precision.

17. The method of any one of claims 1-13, wherein, the artificial intelligence model comprises a hybrid expert architecture large model.

18. The method of any one of claims 1-13, wherein, the weight matrix of the plurality of layers after the quantization operation satisfies one or more of: a memory footprint of the weight matrix of the plurality of layers after the quantization operation is lower than a memory footprint of the weight matrix of the plurality of layers before the quantization operation; a transmission bandwidth of the weight matrix of the plurality of layers after the quantization operation is lower than a transmission bandwidth of the weight matrix of the plurality of layers before the quantization operation; and a running speed of the artificial intelligence model with the weight matrix of the plurality of layers after the quantization operation is higher than a running speed of the artificial intelligence model with the weight matrix of the plurality of layers before the quantization operation. comprise:

19. An electronic device, comprising: at least one processing unit; at least one memory including one or more computer program modules, wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processing unit, the one or more computer program modules being used to implement the method of any one of claims 1-18. instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method of any one of claims 1-18.

20. A computer-readable storage medium, characterized in that, comprise:

21. A rotation-based quantization apparatus for an artificial intelligence model, comprising: a rotation matrix obtaining module configured to obtain a rotation matrix used to perform a rotation operation, wherein the rotation operation comprises a left rotation operation and a right rotation operation; a weight matrix obtaining module configured to obtain a weight matrix of a plurality of layers of the artificial intelligence model; a rotation module configured to use the rotation matrix to perform one of a left rotation operation and a right rotation operation on the weight matrix of the plurality of layers according to a type of the plurality of layers; and a quantization module configured to perform a quantization operation on the weight matrix of the plurality of layers after the rotation operation. the electronic device of claim 19 or the quantization device of claim 21.

22. An electronic device, comprising: ​

Images