Inference operation method and related equipment and system

Abstract

The embodiment of the invention provides an inference operation method and related equipment and system. According to the embodiment of the invention, when the large model is quantized, the learned reversible matrix H is inserted at the matrix multiplication operation position in the large model so as to suppress the abnormal value in activation, and the quantization difficulty of activation is overcome by utilizing the effect that the reversible matrix has a relatively good abnormal value smoothing effect; and the input and weight of the matrix multiplication operation are transformed once by using the learned reversible matrix so as to ensure the invariance of the calculation. In the embodiment of the invention, the reversible matrix fused into the large model is learnable, can adapt to large model quantization, realizes error minimization before and after model quantization, and can improve the model precision after model quantization.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to reasoning and computation methods and related devices and systems. Background Technology

[0002] Many large language models (LLMs), exemplified by ChatGPT, have achieved remarkable success in the field of natural language processing (NLP). These large models have made groundbreaking progress in various tasks such as language translation, text summarization, and question answering. However, these models typically have billions or even hundreds of billions of parameters, requiring massive amounts of storage and computing resources, resulting in high practical deployment costs and making them difficult to deploy on resource-constrained devices such as mobile phones. Summary of the Invention

[0003] Firstly, embodiments of this application provide a reasoning operation method that can perform reasoning operations on the input of an artificial intelligence program based on a quantized fusion large model to obtain reasoning results, effectively ensuring the accuracy of the reasoning operation.

[0004] The method may include: running a first artificial intelligence program; reading a first model file from a first storage space, loading the invertible matrix H stored in the first model file and the quantized first artificial intelligence model into memory; and performing inference operations on the input of the first artificial intelligence program using the quantized first artificial intelligence model loaded into memory, wherein, during the inference operations, the input of the matrix multiplication operation is transformed using the invertible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0005] In the first aspect, the first artificial intelligence model can be the fused large model in subsequent embodiments, which can be obtained by fusing the invertible matrix H into the second artificial intelligence model. The second artificial intelligence model can be the large model before fusion mentioned in subsequent embodiments. The first artificial intelligence program provides artificial intelligence services based on the first artificial intelligence program. The invertible matrix H in the first model file can be used to perform invertible transformations on the input of the matrix multiplication operation part in the first artificial intelligence model to ensure computational invariance.

[0006] The method provided in the first aspect can be executed by terminal devices such as mobile phones and tablets. The quantized first artificial intelligence model not only has a smaller number of model parameters, but also maintains high accuracy after quantization due to the incorporation of an invertible matrix. It is suitable for deployment on resource-constrained terminal devices and can provide high inference accuracy. Of course, the method provided in the first aspect can also be executed by cloud servers, which can reduce cloud resource consumption while ensuring model accuracy.

[0007] In the first aspect, before running the first artificial intelligence program, the first artificial intelligence model can be constructed by fusing the invertible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model. The first artificial intelligence program can provide artificial intelligence services to users based on the second artificial intelligence model. The fusing process can include transforming the weights of matrix multiplication operations in the second artificial intelligence model using the invertible matrix H. This allows the learned invertible matrix H to be inserted at the matrix multiplication positions in the large model during quantization to suppress outliers in activation and overcome the difficulty of activation quantization.

[0008] In conjunction with the first aspect, in some embodiments, the second artificial intelligence model may include L layers, where L is a positive integer and L > 2. The invertible matrix H may include L invertible matrices, each belonging to one of the L layers. The weights of matrix multiplication operations in the second artificial intelligence model are transformed using the invertible matrix H of each layer. Specifically, this can include: using the invertible matrix H of each layer in a layer-by-layer, parallel manner to transform the weights of matrix multiplication operations in each layer. Thus, by performing model fusion in a layer-by-layer, parallel manner, the efficiency of model fusion can be improved.

[0009] The steps described above for constructing the first AI model can be executed by a cloud-based server. Once the first AI model is constructed, the cloud-based server can send the model parameters (including weights) to the terminal device, thus reducing the computational requirements on the terminal device. For terminal devices with strong computational capabilities, this step can also be executed by the terminal device itself.

[0010] In conjunction with the first aspect, in some embodiments, before running the first artificial intelligence program, the method may further include: quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model; and then saving the invertible matrix H and the quantized first artificial intelligence model to a first model file.

[0011] Quantizing the first artificial intelligence model can specifically include quantizing the transformed weights of matrix multiplication operations in each layer to obtain the quantized weights of matrix multiplication operations in each layer. The quantized weights of the first artificial intelligence model can include the quantized weights of matrix multiplication operations in each layer. In this way, by performing quantization in parallel across multiple layers, the efficiency of model quantization can be improved.

[0012] The above steps for quantifying the first artificial intelligence model involve a large amount of computation and can be executed on the cloud or on terminal devices with strong computing power. This application embodiment does not limit this.

[0013] In conjunction with the first aspect, in some embodiments, the invertible matrix H used to be fused into the second artificial intelligence model is learned. Thus, the invertible matrix fused into the second artificial intelligence model is learnable, capable of adapting to large model quantization, minimizing the error before and after model quantization, and improving the accuracy of the quantized model.

[0014] The specific implementation of learning can include:

[0015] The invertible matrix H of the i-th layer of the first artificial intelligence model is learned by minimizing the following objective function using gradient descent. i :

[0016]

[0017] Among them, f i This represents the output of the i-th layer of the first AI model before fusion. L represents the output of the i-th layer of the first AI model after fusion. i Indicates output With output f i The distance.

[0018] In conjunction with the first aspect, in some embodiments, the invertible matrices of each layer of the first artificial intelligence model can be learned separately using different computing devices, including graphics processing units (GPUs). This improves the learning speed of the invertible matrices, thereby increasing model compression efficiency, and reduces the computational burden on a single GPU, as the computing power of a single GPU only needs to be sufficient to handle the learning of a single-layer invertible matrix.

[0019] The steps for learning the invertible matrix described above can be executed on the cloud side or on a terminal device with strong computing power. This application embodiment does not impose any restrictions on this.

[0020] In conjunction with the first aspect, in some embodiments, a larger invertible matrix H can also be decomposed into the Kronecker product of two smaller invertible matrices H1 and H2 to reduce the inference latency of large models. That is: invertible matrix H = H1 * H2, where H1 and H2 are also invertible matrices, and "*" represents the Kronecker product of the two matrices.

[0021] Secondly, embodiments of this application provide a model quantization method applicable to cloud-side servers. The method may include: the cloud-side server fusing an invertible matrix H into a second artificial intelligence model to obtain a first artificial intelligence model; the cloud-side server quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model; the cloud-side server saving the invertible matrix H and the quantized first artificial intelligence model into a first model file; and the cloud-side server sending the first model file to a terminal device.

[0022] Implementing the second approach can reduce the number of model parameters on the terminal device side through quantization, enabling resource-constrained terminal devices to perform inference operations using large models. Moreover, by incorporating an invertible matrix H into the large model during quantization, outliers in activation are suppressed, reducing the difficulty of model quantization and effectively preserving model accuracy. This, in turn, improves the accuracy of inference operations performed by the terminal device using the quantized large model.

[0023] In conjunction with the second aspect, in some embodiments, the invertible matrix H is integrated into the second artificial intelligence model. Specifically, this may include: the cloud-side server using the invertible matrix H to transform the weights of matrix multiplication operations in the second artificial intelligence model.

[0024] In conjunction with the second aspect, in some embodiments, the second artificial intelligence model may include L layers, where L is a positive integer and L > 2. The invertible matrix H may include L invertible matrices, each belonging to one of the L layers. The weights of matrix multiplication operations in the second artificial intelligence model are transformed using the invertible matrix H. Specifically, this can be achieved by the cloud-side server using the invertible matrix H of each layer in a layered and parallel manner to transform the weights of matrix multiplication operations in each layer. In this way, by performing model fusion in a layered and parallel manner, the efficiency of model fusion can be improved.

[0025] In conjunction with the second aspect, in some embodiments, quantization of the first artificial intelligence model may specifically include: the cloud-side server quantizing the transformed weights of matrix multiplication operations in each layer to obtain the quantized weights of matrix multiplication operations in each layer. The quantized weights of the first artificial intelligence model may include the quantized weights of matrix multiplication operations in each layer. Thus, by performing quantization in parallel across multiple layers, the efficiency of model quantization can be improved.

[0026] In conjunction with the second aspect, in some embodiments, the invertible matrix H used to fuse into the second artificial intelligence model is obtained through learning. Thus, the invertible matrix fused into the second artificial intelligence model is learnable, capable of adapting to large model quantization, minimizing errors before and after model quantization, and improving the accuracy of the quantized model.

[0027] The specific implementation of learning can include:

[0028] The cloud-based server learns the invertible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function using gradient descent. i :

[0029]

[0030] Among them, f i This represents the output of the i-th layer of the first AI model before fusion. L represents the output of the i-th layer of the first AI model after fusion. i Indicates output With output f i The distance.

[0031] In conjunction with the second aspect, in some embodiments, the cloud-side server can learn the invertible matrices of each layer of the first artificial intelligence model using different computing devices, including graphics processing units (GPUs). This allows the cloud-side server to improve the learning speed of the invertible matrices, thereby increasing model compression efficiency, and reducing the computational burden on a single GPU, as the computing power of a single GPU only needs to be sufficient to handle the learning of a single layer of the invertible matrix.

[0032] In conjunction with the second aspect, in some embodiments, the cloud-side server can also decompose a large invertible matrix H into the form of a Kronecker product of two smaller invertible matrices H1 and H2 to reduce the inference latency of large models. That is: the invertible matrix H = H1 * H2, where H1 and H2 are also invertible matrices, and "*" represents the Kronecker product of the two matrices.

[0033] Thirdly, embodiments of this application provide an inference operation method applicable to terminal devices. The method may include: receiving a first model file sent by a cloud-side server; saving the first model file to a first storage space; running a first artificial intelligence program; reading the first model file from the first storage space; loading the invertible matrix H stored in the first model file and the quantized first artificial intelligence model into memory; and performing inference operations on the input of the first artificial intelligence program using the quantized first artificial intelligence model loaded into memory. Specifically, during the inference operation, the input to the matrix multiplication operation is transformed using the invertible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0034] In the third aspect, the first artificial intelligence model can be obtained by fusing an invertible matrix H into a second artificial intelligence model. The first artificial intelligence program provides artificial intelligence services based on the first artificial intelligence program. The invertible matrix H in the first model file can be used to perform invertible transformations on the input of the matrix multiplication operation in the first artificial intelligence model to ensure computational invariance.

[0035] Implementing the third approach allows for reasoning operations on the input of artificial intelligence programs based on the quantified fusion model, yielding reasoning results and effectively ensuring the accuracy of the reasoning operations.

[0036] In conjunction with the third aspect, in some embodiments, the terminal device may run the first artificial intelligence program after detecting a user operation that runs the first artificial intelligence program. This user operation may be, for example, clicking the desktop icon of the first artificial intelligence program, or navigating to the first artificial intelligence program from another application. The embodiments of this application do not limit the implementation of this operation. Not limited to running the first artificial intelligence program in response to a user operation, the terminal device may also run the first artificial intelligence program based on internal events, such as activating a smart voice assistant when the remaining battery power is high to provide companionship and conversation. The embodiments of this application do not limit the conditions for launching the first artificial intelligence program.

[0037] In conjunction with the third aspect, in some embodiments, the first artificial intelligence program may also be a resident program on the terminal device that starts running from the moment the device is powered on.

[0038] Fourthly, embodiments of this application provide an electronic device that may include multiple functional modules that cooperate with each other to implement the method described in the first aspect or any possible implementation of the first aspect.

[0039] Fifthly, embodiments of this application provide an electronic device that may include: a memory and a processor, the memory being coupled to the processor, a computer program being stored in the memory, and the computer program being executed by the processor to implement the method described in the first aspect or any possible implementation of the first aspect.

[0040] Sixthly, embodiments of this application provide a server that may include multiple functional modules that cooperate with each other to implement the method described in the second aspect or any possible implementation of the second aspect.

[0041] In a seventh aspect, embodiments of this application provide a server that may include: a memory and a processor, the memory being coupled to the processor, a computer program being stored in the memory, and the computer program being executed by the processor to implement the method described in the second aspect or any possible implementation of the second aspect.

[0042] Eighthly, embodiments of this application provide a terminal device, the server of which may include multiple functional modules that cooperate with each other to implement the method described in the third aspect or any possible implementation of the third aspect.

[0043] Ninthly, embodiments of this application provide a terminal device that may include: a memory and a processor, the memory being coupled to the processor, a computer program being stored in the memory, and the computer program being executed by the processor to implement the method described in the third aspect or any possible implementation of the third aspect.

[0044] In a tenth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or any possible implementation of the first aspect, or implements the method described in the second aspect or any possible implementation of the second aspect, or implements the method described in the third aspect or any possible implementation of the third aspect.

[0045] In one aspect, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation of the first aspect, or implements the method described in the second aspect or any possible implementation of the second aspect, or implements the method described in the third aspect or any possible implementation of the third aspect. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0047] Figure 1 An edge-cloud model deployment system 10 provided in an embodiment of this application is shown;

[0048] Figure 2 The comparison shows the outliers activated during existing simple quantization and those activated during Quarot quantization;

[0049] Figure 3 A simplified structure of the large model in an embodiment of this application is shown;

[0050] Figure 4 The overall flow of the reasoning operation method provided in the embodiments of this application is shown;

[0051] Figure 5 The terminal device 30 provided in an embodiment of this application is shown;

[0052] Figure 6 The server 20 provided in an embodiment of this application is shown;

[0053] Figure 7 An electronic device 300 provided in an embodiment of this application is shown. Detailed Implementation

[0054] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0055] Figure 1 An edge-cloud model deployment system 10 provided in an embodiment of this application is shown.

[0056] like Figure 1 As shown, the edge-cloud model deployment system 10 may include: a server 20 and terminal devices 30. A communication link may be established between the server 20 and the terminal devices 30.

[0057] Server 20 can be used to train large models such as Large Language Models (LLMs) using datasets. Well-trained large models can provide various artificial intelligence (AI) functions such as language translation, text summarization, and question answering. However, well-trained large models typically have billions or even hundreds of billions of parameters, requiring massive storage and computing resources, making them difficult to deploy on resource-constrained devices such as mobile phones.

[0058] Server 20 can also be used to compress large language models (LLMs) through methods such as pruning, quantization, and semi-structured sparsity. Among them, quantization reduces the numerical precision of model parameters and intermediate activations, such as converting 32-bit floating-point numbers to 8-bit or 4-bit integers, thereby reducing the parameter storage requirements of the model and speeding up the calculation, making it easier to deploy large models on terminal devices 30 such as mobile phones.

[0059] Server 20 can also be used to transmit the quantized model parameters to terminal device 30 after quantization is completed. Generally, the quantized model parameters may include quantization weights, such as weights with lower numerical precision.

[0060] Terminal device 30 can obtain quantized model parameters from server 20, load the quantized model parameters, and then use the quantized model parameters to perform inference operations on user inputs such as search terms and text to be translated to obtain inference results. Here, loading the quantized model parameters by terminal device 30 can mean loading the quantized model parameters, such as quantized weights, into the memory of terminal device 30.

[0061] In this embodiment, the server 20 can use a quantization method to compress the large model to achieve model miniaturization, and deploy the model on terminal devices 30 such as mobile phones, tablets, smart screen devices, virtual reality (VR) devices, augmented reality (AR) devices, etc., thereby providing users with high-quality text summarization, context-aware intelligent reply, advanced grammar correction and other artificial intelligence functions or services on the terminal devices 30.

[0062] Quantization can include quantization-aware training (QAT) and post-training quantization (PTQ). Compared to QAT, PTQ does not require retraining the model, nor does it require modification of the model architecture. It has no requirements on the dataset or training environment, making it highly versatile and an efficient and practical model compression method. The core idea of ​​PTQ is to quantize the model's weights and activations using a small amount of calibration data or no data after training a large model, thereby significantly reducing the storage requirements and computational overhead of the large model. The quantization objects of PTQ can include weights and activations. Activations are more difficult to quantize than weights because weights are generally more uniformly distributed, while activations contain a large number of outliers. The presence of outliers greatly increases the difficulty of quantization, making it difficult for simple quantization strategies to retain both outliers and non-outliers using the same quantization parameters after quantization, especially in low-bit quantization scenarios, ultimately leading to poor model compression results.

[0063] Even though activations are more difficult to quantize than weights due to outliers, different tokens exhibit similar changes across their channels, with outliers typically appearing on fixed channels. Based on this observation, the SmoothQuant quantization method was proposed. It balances the quantization difficulty of weights and activations by migrating outliers from activations to weights through equivalent scaling between channels. Further reference to the SmoothQuant method can be found in: Guangxuan Xiao, et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models", International Conference on Machine Learning, 2023, the entire contents of which are incorporated herein by reference.

[0064] The OmniQuant quantization method, building upon SmoothQuant, uses learnable scaling and offset coefficients to smooth and suppress outliers in activations. It proposes a quantization method that learns the scaling and offset coefficients through training and a small amount of calibration data to obtain the optimal scaling and offset coefficients. Further reference to the OmniQuant method can be found in the following literature: Wenqi Shao, et al., "OMNIQUANT: OMNIDIRECTIONALLY CALIBRATED QUANTIZATION FOR LARGE LANGUAGE MODELS", The International Conference on Learning Representations (ICLR), 2024, the entire contents of which are incorporated herein by reference.

[0065] However, SmoothQuant and OmniQuant only balance the difficulty of quantizing weights and activations, changing the size of outliers through vector multiplication. The overall quantization difficulty of large models is still very high, especially in low-precision quantization scenarios such as 4 bits, where they cannot effectively preserve the capabilities and accuracy of large models.

[0066] The QuaRot quantization method proposes to use a randomized Hadamard matrix to perform a Hadamard transformation on the model weights to suppress outliers, making activation easier to quantize. Figure 2 The comparison shows the outliers activated during simple quantization and those activated during QuaRot quantization. For example... Figure 2 As shown, when performing existing simple quantization on large models, outliers exist in the activations, which deviate significantly from other values, leading to high quantization difficulty. However, with QuaRot quantization, outliers in the activations can be eliminated, reducing quantization difficulty. However, the QuaRot quantization method uses the same Hadamard transformation for outlier smoothing in all matrix multiplications, failing to utilize the patterns of outlier occurrence and thus not effectively preserving model capabilities and accuracy. Further reference to the QuaRot method can be found in the following literature: Saleh Ashkboos, et al., "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs", The International Conference on Learning Representations (ICLR), 2024, the entire contents of which are incorporated herein by reference.

[0067] To effectively preserve model accuracy and capability while suppressing outliers in activation, in this embodiment, server 20 can first insert learned invertible matrices at the matrix multiplication positions in the large model for model fusion, and then quantize the fused large model. In this way, the learned invertible matrix H can adapt to different outlier occurrence patterns, effectively suppressing outliers in activation, reducing quantization difficulty, minimizing the error before and after model quantization, and preserving the capability and accuracy of the large model.

[0068] Table 1 below illustrates the impact of several quantization methods on model perplexity. "LLaMA-2-7B" represents a large language model with seven billion (7B) parameters; "WikiText-2perplexity" represents the model's perplexity on the "WikiText-2" test dataset; and "C4perplexity" represents the model's perplexity on the "C4" test dataset. "FP16" indicates that the model is unquantized and uses 16-bit floating-point high precision. "SmoothQuant" and "OminiQuant" represent the aforementioned SmoothQuant and OmniQuant quantization methods, respectively. "AQuant" represents the quantization method based on the learned invertible matrix provided in this embodiment. As shown in Table 1, in the performance of "WikiText-2 perplexity", the perplexity of the unquantized model (FP16) is relatively low at 5.48, indicating strong model capability and good language prediction performance. After using SmoothQuant quantization, the perplexity of the model increases significantly to 83.12, indicating a significant decrease in predictive ability. Using OminiQuant quantization also results in a relatively high perplexity of 14.26, indicating poor model capability. However, using "AQuant" quantization results in a perplexity of 5.77, close to that of "FP16", demonstrating that the model's capability is effectively preserved. Similarly, in the performance of "C4 perplexity", compared to other quantization methods for suppressing outliers, the "AQuant" quantization method brings the model's perplexity closer to that of "FP16", better preserving the model's capability.

[0069] LLaMA-2-7B WikiText-2perplexity C4 perplexity FP16 5.48 7.28 SmoothQuant 83.12 77.49 OmniQuant 14.26 18.53 AQuant 5.77 7.84

[0070] Table 1

[0071] Table 2 below shows the impact of several quantization methods on model accuracy. Among them, "LLaMA3-8B" represents a large language model with eight billion (8B) parameters, and "ARC-Challenge", "ARC-Easy", "HellaSwag", "LAMBADA", "PIQA", etc.

[0072] “WinoGrande” represents several different question-and-answer (Q&A) test datasets. “FP16” indicates that the model is not quantized, “QuaRot” indicates the aforementioned QuaRot quantization method, and “AQuant” indicates the quantization method based on the learned invertible matrix provided in this application embodiment. As can be seen from Table 2, in terms of performance on several different question-and-answer (Q&A) test datasets, compared to the model accuracy quantized by “QuaRot”, the model accuracy quantized by “AQuant” is closer to the accuracy of the unquantized model (FP16), thus better preserving the model accuracy.

[0073] LLaMA3-8B ARC-Challenge ARC-Easy HellaSwag LAMBADA PIQA WinoGrande FP16 54.01 77.65 79.21 72.16 80.41 73.40 QuaRot 45.73 70.83 72.97 62.70 75.35 67.17 AQuant 49.83 77.65 76.96 68.92 78.07 70.72

[0074] Table 2

[0075] In addition to incorporating the quantized weights of the large model, the quantized model parameters transmitted by server 20 to terminal device 30 can further include an invertible matrix H. After receiving the quantized model parameters, terminal device 30 can first transform and quantize the input using the invertible matrix H at the matrix multiplication operations in the large model, and then perform matrix multiplication operations using the transformed and quantized input and the quantized weights transformed by the invertible matrix H to achieve model computation invariance.

[0076] In large models, the learning of the invertible matrix H is performed hierarchically. Different layers can learn their respective invertible matrices in parallel, and these matrices can be distributed across different computing devices (such as graphics processing units, GPUs) for parallel learning, which improves model compression efficiency. How the invertible matrix H is learned will be explained in detail in subsequent embodiments and will not be elaborated upon here.

[0077] This application provides a reasoning operation method that can be applied to... Figure 1 An exemplary edge-cloud model deployment system 10 is shown. This inference operation method can be implemented based on the quantized fused large model, and the precision and accuracy of its inference operation are effectively guaranteed by the quantized fused large model.

[0078] To effectively preserve model accuracy, the main concept of model quantization can include: learning to find an invertible matrix that effectively smooths outliers; inserting the learned invertible matrix H at matrix multiplication points in the large model during quantization (i.e., implementing model fusion) to suppress outliers in activations, reducing quantization difficulty; and utilizing the inverse property of the invertible matrix and its inverse to ensure computational invariance. Since the invertible matrix is ​​learned, it can adapt to the patterns of outlier occurrence in large model quantization, minimizing errors before and after quantization and preserving model capabilities after quantization. This will be elaborated below.

[0079] 1. Learned invertible matrices can be incorporated into large models.

[0080] like Figure 2 As shown, the large model in this embodiment can be, for example, a Transformer model, which can be divided into multiple layers, such as layer 1, layer 2, etc. The output of the previous layer in the large model can be used as the input of the next layer; for example, the output f1 of layer 1 is the input X1 of layer 2. Each layer in the large model can include the same sublayers, one of which is self-attention, and the other is a feed-forward network (FFN). Within the same layer, the output of self-attention can be used as the input of the feed-forward network.

[0081] Ignoring other sublayers, such as the addition and normalization sublayers (Add & Norm), the output f of the i-th layer... i This can be simply represented by the following formula:

[0082] f i =FFN i (SelfAttention i (x i ))(one)

[0083] Where, x i Represents the input of the i-th layer, SelfAttention i FFN represents the operation performed by the self-attention layer i. i This represents the operation performed by the feedforward network (FFN) in the i-th layer. i (SelfAttention i (x i The )) indicates that the result of the self-attention operation can be used as the input of the feedforward network.

[0084] The self-attention mechanism in each layer can include four linear sub-layers: Query layer, Key layer, Value layer, and Output layer. i (x i This can be expressed by the following formula:

[0085]

[0086] in, These represent the weights of the Query layer, Key layer, Value layer, and Output layer in the self-attention layer of the i-th layer, respectively. express.......

[0087] about FFN i The formula expansion can be found in the following reference: Ashish Vaswani*, et al., "Attention Is All You Need", Advances in neural information processing systems, 2017, the entire contents of which are incorporated herein by reference. The single-layer algorithm flow of large models can also be implemented in other forms, and the embodiments of this application do not limit this.

[0088] f calculated by the above formula (I) i It is the output of the i-th layer of the large, unquantized model, which has a large computational load, resulting in low model inference efficiency.

[0089] Matrix multiplication is widely used in large models like the Transformer, especially in the linear layers of these models. Linear layers are further widely used in self-attention and feedforward neural networks across various layers of these large models, serving to adjust dimensionality and transform features, thus providing the model with better representation and fitting capabilities.

[0090] In this embodiment, server 20 can quantize matrix multiplication operations in a large model to achieve large model quantization, thereby reducing the storage space required by the large model and improving the model inference efficiency; furthermore, server 20 can also transform the input and weights of matrix multiplication operations during quantization, which can eliminate the influence of outliers in activation on quantization and ensure the accuracy of the quantized model.

[0091] There are locations in the self-attention sublayer of layer i where matrix multiplication occurs, such as in the above formula (ii). This refers to matrix multiplication. Matrix multiplication can also be included in the i-th layer of the feedforward network. The embodiments of this application use... This represents the output of the i-th layer of the large model after quantization matrix multiplication.

[0092] Without considering other sub-layers that may be included in each layer, such as addition and normalization sub-layers (Add & Norm), server 20 can determine the output of the i-th layer of the large model after quantized matrix multiplication using the following simplified formula.

[0093]

[0094] Where, x i This represents the input of the i-th layer; This indicates the operation performed by the self-attention of the i-th layer. The matrix multiplication operation in the operation undergoes quantization, and during quantization, the inputs and weights involved in the matrix multiplication operation are transformed using invertible matrices. This indicates the operation performed by the feedforward network (FFN) at layer i. The matrix multiplication operation in the operation undergoes quantization, and during quantization, the inputs and weights involved in the matrix multiplication operation are transformed using invertible matrices.

[0095] Server 20 can be calculated using the following formula (IV).

[0096]

[0097] Among them, H i Let H represent the invertible matrix of the i-th layer of the large model. The invertible matrix H is inserted into the linear sub-layers such as the Query layer, Key layer, and Value layer within the self-attention mechanism of the i-th layer. i The specifics can differ.

[0098] Comparing equation (IV) with equation (II), it can be seen that the matrix multiplication operation in the large model is quantized, and the inputs and weights involved in the matrix multiplication operation during quantization are all invertible matrices H. i A transformation was performed.

[0099] For example, in equation (ii) This matrix multiplication operation is a linear matrix multiplication operation performed by the Query layer in the self-attention process, which becomes the operation in equation (iv). Where, x i H i , They can be represented using the invertible matrix H respectively. i The input x involved in this matrix multiplication operation i Weight Perform one transformation on each, int(x) i H i ), This can represent quantizing the transformed input and weights separately, with `int` representing the quantization operation. Similarly, in formula (II)... The matrix multiplication operation can also be quantized after transforming the input and weights. This reduces the difficulty of quantizing large models due to activation quantization.

[0100] While reducing the difficulty of quantization for large models caused by activation quantization, the model quantization method provided in this application can also keep the computational results of large models unchanged, that is, maintain computational invariance. The following uses the smallest matrix multiplication unit Y = XW to illustrate this effect, where Y represents the output of the matrix multiplication operation, X represents the input of the matrix multiplication operation, and W represents the weight of the matrix multiplication operation.

[0101] A simple quantization strategy is applied to quantize matrix multiplication, and its output is labeled as follows: It can be represented as follows:

[0102]

[0103] In this embodiment of the application, the formula (v) used for quantizing matrix multiplication can be converted into:

[0104]

[0105] It can be seen that by utilizing the inverse property of an invertible matrix and its inverse matrix, and performing the learned transformation and its inverse transformation on the input X and weight W respectively, the computational invariance of matrix multiplication can be achieved.

[0106] First, each matrix multiplication operation in the large model can incorporate an invertible matrix. The matrix multiplication operation that incorporates an invertible matrix can be represented as follows: 1. The weights of the matrix multiplication operation undergo a learnable transformation to obtain the transformed weights, which can be seen in formula (VI) under WH. -1 2. The input to the matrix multiplication operation undergoes a learnable transformation to obtain the transformed input, as shown in equation (VI) XH. Then, the large model incorporating invertible matrices can be quantized, specifically including: the transformed input and weights of the matrix multiplication operation can be quantized.

[0107] 2. Layered learning yields the invertible matrices of each layer of the large model.

[0108] In this embodiment of the application, an invertible matrix H is added to each layer of the large model. i It can be learned to adapt to the patterns of outlier occurrences, thereby helping to improve model accuracy.

[0109] For the i-th layer of a large model, server 20 can learn the optimal invertible matrix H of the i-th layer by minimizing the following objective function using the gradient descent algorithm. i :

[0110]

[0111] Among them, fi i This represents the output of the i-th layer of the previous large model. L represents the output of the i-th layer of the fused large model. i express with f i The distance between them. The objective function represented by the above formula (VII) is used to find the invertible matrix H that minimizes this distance. i .

[0112] f i The calculation formula can be found in the aforementioned formulas (I) to (IV), which can be substituted into formula (VII) to expand the objective function.

[0113] Gradient descent specifically works by calculating the gradient of an invertible matrix H with respect to the objective function, iteratively searching for the optimal invertible matrix that minimizes the value of the objective training function. One iteration of gradient descent can be represented by the following algorithm:

[0114]

[0115] Among them, H i(j-1) H represents the invertible matrix of the i-th layer determined by the (j-1)-th iteration of the gradient descent algorithm; ij Let represent the invertible matrix of the i-th layer determined by the j-th iteration of the gradient descent algorithm; γ is a coefficient greater than 0, which can be adjusted in each iteration; Indicates the target training function with respect to H i(j-1) Find the derivative.

[0116] f in the above objective function i It is the output of the i-th layer of the model before fusion. The outputs of each layer of the model before fusion can be calculated in advance using calibration data as model input.

[0117] Server 20 can learn in parallel the optimal invertible matrix H of each layer of a large model. i To improve the efficiency of model compression deployment, the learning of invertible matrices between different layers of a large model can be independent. The learning of the invertible matrix of each layer does not depend on the learning results of the previous layer, and the learning between layers can be executed in parallel. Server 20 can distribute the learning of the invertible matrix of each layer to multiple computing devices, such as multiple GPUs, and learn the invertible matrix of each layer of the large model separately through different computing devices. In this way, the learning speed of invertible matrices can be improved, thereby improving the model compression efficiency, and the computational burden of a single GPU can be reduced, as the computing power of a single GPU only needs to be able to handle the learning of the invertible matrix of a single layer.

[0118] Furthermore, server 20 can also decompose a large invertible matrix H into the form of a Kronecker product of two smaller invertible matrices H1 and H2 to reduce the inference latency of large models.

[0119] Figure 4 The overall flow of the reasoning operation method provided in the embodiments of this application is shown. Figure 4 The terminal devices and servers mentioned above can be the aforementioned Figure 1 The system 10 shown includes terminal device 30 and server 20. The reasoning and calculation method provided in the embodiments of this application will be described below.

[0120] S101, the server determines the invertible matrix H to be fused into the large model.

[0121] In this embodiment, the invertible matrix fused into the large model is learned, which enables the invertible matrix H to minimize the output deviation of each layer before and after model quantization after fusion into the large model, thereby achieving adaptive quantization for the model.

[0122] Specifically, the server can learn in layers to determine the invertible matrices of each layer that are integrated into the large model. For details on how to learn the invertible matrices of each layer, please refer to the previous sections; these will not be repeated here.

[0123] Here, fusion can refer to transforming the weights of matrix multiplication operations in the large model using an invertible matrix H. The invertible matrix H of the i-th layer of the large model... i It can be used to transform the weights of matrix multiplication operations in the i-th layer.

[0124] Invertible matrix H i This can be used to minimize the output deviation of the i-th layer before and after quantization. For the i-th layer of a large model, the server can train the invertible matrix H of the i-th layer using the aforementioned objective function. i Specifically: First, the server can input the text from a small amount of calibration data into the large model. During training, this input text can be used as the input X1 of the first layer, resulting in the unquantized matrix output f of each layer of the large model. i The output f of the i-th layer i The input x of the (i+1)th layer i+1 Then, the server can perform unquantized matrix multiplication on the output f of each layer. i Weight W i (such as the weights of the four linear layers in self-attention) The data is distributed to multiple computing devices, such as multiple GPUs, and these devices are controlled to work in parallel to obtain the invertible matrix H of each layer through the aforementioned objective function. i .

[0125] Calculate the output f of each layer before quantization matrix operation. i The formulas can be found in formulas (I) and (II) mentioned above, and will not be repeated here. Calculate the output of each layer after quantization matrix operations. The formulas can be found in formulas (III) and (IV) above, and will not be repeated here.

[0126] The hierarchical parallel approach uses the aforementioned objective function to find the invertible matrix that minimizes the error before and after quantization. The process of finding the invertible matrix in each layer does not depend on the result of the previous layer, which can significantly improve the model compression efficiency and reduce the computational power requirements of a single computing device.

[0127] The calibration data can contain multiple text inputs, which can be used as training data for multiple training sessions to obtain a more adaptive invertible matrix.

[0128] S102, the server merges the invertible matrix H with the weights W of the large model, that is, it merges the invertible matrix H into the large model to obtain the merged large model.

[0129] Fusion can refer to transforming the weights of matrix multiplication operations in a large model using an invertible matrix H. Specifically, S102 can be implemented as follows: The server uses the invertible matrix H to transform the weights W of the matrix multiplication operations in the large model, obtaining the transformed weights, which are the weights of matrix multiplication operations in the fused large model, as shown in the aforementioned formula (VI) WH. The weights of matrix multiplication operations in the fused large model have undergone a transformation compared to the weights of matrix multiplication operations in the large model before fusion.

[0130] Assume the large model consists of L layers, where L is a positive integer and L > 2. Then the invertible matrices incorporated into the large model can include L invertible matrices: H0, H1, ..., H... L These L invertible matrices belong to L layers. To improve fusion efficiency, S102 can be executed in a layered parallel manner in the large model. That is, the weights of matrix multiplication operations in each layer are transformed using the invertible matrices of each layer in a layered parallel manner. For the i-th layer of the large model, the server can use the invertible matrix H of the i-th layer. i The weight W of the matrix multiplication part in the i-th layer i Perform the transformation, that is, use the invertible matrix H i Multiplied by weight W i To realize an invertible matrix H i The i-th layer is integrated into the larger model.

[0131] S103, the server quantizes the fused large model.

[0132] The server quantizes the fused large model, which may include: the server quantizes the weights of matrix multiplication operations in the fused large model to obtain the quantized weights of matrix multiplication operations, which can be referred to in the aforementioned formula (VI) in int(WH -1 The weights of the quantized fusion model (or the quantized weights of the fusion model) may include the quantized weights of matrix multiplication operations in the fusion model.

[0133] S103 can also be performed layer by layer in large models to improve quantization efficiency. That is, the server can perform parallel quantization of the weights of matrix multiplication operations in each layer of the large model to obtain the quantized weights of matrix multiplication operations in each layer, such as int(W1H1...). -1 ), int(W2H2) -1), ..., int(W L H L -1 int(W1H1) -1 ), int(W2H2) -1 ..., int(W L H L -1 ) represent the quantization weights of matrix multiplication operations in layers 1, 2, ..., L of the fusion model, respectively.

[0134] The matrix multiplication operation using the quantization method provided in this application embodiment in a certain layer can be one or more matrix multiplication operations in that layer, or all matrix multiplication operations. The quantization method provided in this application embodiment can also occur only in some layers of a large model, without being forced to occur in all layers of the large model.

[0135] The above S101-S103 can be used to realize large model fusion and model quantization, preparing for subsequent model inference operations.

[0136] S104, the server transmits the first model file to the terminal device. The first model file includes an invertible matrix H and a quantized fused large model. Correspondingly, the terminal device can receive the first model file.

[0137] The quantized fusion model can include: the quantization weights of the fusion model, specifically the quantization weights of each layer of the fusion model. The quantization weights of each layer can include the quantization weights of matrix multiplication operations within that layer; for example, the quantization weights of layer 1 could include int(W1H1...). -1 The quantization weights of the second layer can include int(W2H2). -1 ), int(W1H1 -1 ), int(W2H2) -1 ) represent the quantization weights of matrix multiplication operations in the first and second layers of the fusion model, respectively.

[0138] S105, the terminal device saves the first model file to the first storage space.

[0139] Alternatively, the server can send the download addresses of the quantized fusion model and the invertible matrix H to the terminal device instead of directly transmitting the invertible matrix H and the quantized fusion model to the terminal device. The terminal device can then download the quantized fusion model and the invertible matrix H from the cloud server hosting the fusion model and the invertible matrix H based on the download addresses, and save them to the first model file.

[0140] S106, the terminal device runs the first artificial intelligence program.

[0141] The first artificial intelligence program can be a system application or a third-party application on a terminal device that can provide artificial intelligence services, or it can be an artificial intelligence service provided by a system application or a third-party application.

[0142] The first artificial intelligence program can provide artificial intelligence services based on a large model. The reasoning results can be obtained by running the first artificial intelligence model to perform reasoning operations on the input of the first artificial intelligence program.

[0143] The terminal device can run the first artificial intelligence program after detecting a user action that runs the first artificial intelligence program. This user action could be, for example, clicking the desktop icon of the first artificial intelligence program, or navigating to the first artificial intelligence program from another application. This application embodiment does not limit the implementation of this action. It is not limited to running the first artificial intelligence program in response to a user action; the terminal device can also run the first artificial intelligence program based on internal events, such as activating a smart voice assistant when the remaining battery power is high to provide companionship and conversation. This application embodiment does not limit the conditions for starting the first artificial intelligence program.

[0144] The first artificial intelligence program can also be a resident program on the terminal device, which starts running when the device is powered on.

[0145] S107, the terminal device reads the first model file from the first storage space and loads the invertible matrix H and the quantized fused large model stored in the first model file into memory.

[0146] The terminal device loads the quantized fused large model into memory, which may include: the terminal device loading the quantization weights of the fused large model (including the quantization weights of matrix multiplication operations in the fused large model) into memory.

[0147] S108, the terminal device can receive user input X through the first artificial intelligence program. in , such as the text to be translated.

[0148] S109, in response to the user input X in The terminal device can use the quantized fusion model to analyze the user input X. in Inference operations are performed, whereby the input to the matrix multiplication operation is transformed using an invertible matrix H before the matrix multiplication operation in the quantized fusion model.

[0149] The merged large model (i.e., the merged large model) and the pre-merger large model can be referred to as the first artificial intelligence model and the second artificial intelligence model, respectively.

[0150] by Figure 3Taking the illustrated Transformer model as an example, in the quantized fusion model, the inference operation can be performed layer by layer as follows:

[0151] In the first layer of the fusion model:

[0152] S1-1. The terminal device can first transform the user input X1 of the first layer using the invertible matrix H1 of the first layer, and then quantize the transformed user input to obtain int(x1H1). X1 = X in .

[0153] S1-2. Then, the terminal device calculates the output of the first layer based on the quantized model structure of the first layer of the fusion big model and int(x1H1).

[0154] in, These represent the quantization weights of several matrix multiplication operations in the first layer of the fusion model. These matrix multiplication operations are located in the Query, Key, Value, and Output layers of the self-attention mechanism in the first layer. The first layer is not limited to these matrix multiplication operations in the self-attention mechanism; it can also include matrix multiplication operations distributed throughout the FFN.

[0155] In the second layer of the fusion model:

[0156] S2-1. The terminal device can first use the invertible matrix H2 of the second layer to transform the user input X2 of the second layer, and then quantize the transformed user input to obtain int(x2H2).

[0157] S2-2. Then, the terminal device calculates the output of the second layer based on the quantized model structure of the second layer of the fusion big model and int(x2H2).

[0158] in, These represent the quantization weights of several matrix multiplication operations in the second layer of the fusion model. These matrix multiplication operations are located in the Query, Key, Value, and Output layers of the self-attention mechanism in the second layer. The second layer is not limited to these matrix multiplication operations in the self-attention mechanism; it can also include matrix multiplication operations distributed throughout the FFN.

[0159] And so on, in the i-th layer of the fusion large model (2 < i < L, where L is the number of layers in the large model):

[0160] Si-1, the terminal device can first utilize the invertible matrix H of the i-th layer. iFor the user input X at level i i Perform the transformation, then quantize the transformed user input to obtain int(x) i H i ). That is, the input of each layer is equal to the output of the layer above.

[0161] Si-2, then, the terminal device is based on the quantized model structure of the i-th layer of the fusion large model and int(x) i H i Calculate the output of the i-th layer.

[0162] in, These represent the quantization weights of several matrix multiplication operations in the i-th layer of the fusion model. These matrix multiplication operations are located in the Query, Key, Value, and Output layers of the self-attention mechanism in the i-th layer. The i-th layer is not limited to these matrix multiplication operations in the self-attention mechanism; it can also include matrix multiplication operations distributed throughout the FFN.

[0163] Based on the above reasoning and calculation process, it can be concluded that S108 may include: at the matrix multiplication operation positions in each layer of the fusion large model, the terminal device can first utilize the invertible matrix H of each layer. i Input x for matrix multiplication operations at each layer i The input is transformed, then quantized, and then the quantized transformed input (which can be represented as int(x)) is obtained through matrix multiplication operations in each layer. i H i The quantization weights of matrix multiplication operations in each layer (which can be represented as int(W)) and the quantization weights of matrix multiplication operations in each layer (which can be represented as int(W)) i H i -1 i ),like Perform matrix multiplication.

[0164] Figure 4 The illustrated process can also be executed on a single device, such as a server or a terminal device with strong storage and computing power. If executed on a single device, step S104 in the above process can be omitted. (Subsequent...) Figure 7 The electronic device shown in the embodiments can be one implementation of the single device.

[0165] Not limited to large models, the embodiments of this application can also be applied to models with other numbers of parameters, and smaller models can also be quantized using the methods provided in the embodiments of this application.

[0166] Figure 5The terminal device 30 provided in an embodiment of this application is shown.

[0167] Terminal device 30 can be any of the terminal devices mentioned in the foregoing embodiments. The device type of terminal device 30 can be any of the following: mobile phone, tablet computer, handheld computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), as well as smart home devices such as smart screens and smart speakers, wearable devices such as smart bracelets, smartwatches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices, or smart city devices, etc.

[0168] like Figure 5 As shown, the terminal device 30 may include: a processor 110, a memory 120, a display screen 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180B, an accelerometer sensor 180E, and a touch sensor 180K, etc. The various components in the terminal device 30 can be connected via a bus.

[0169] The processor 110 can be one or more, and they can be integrated into an integrated circuit of a system-on-a-chip (SOC). An SOC is a system-on-a-chip. The processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and a display driver integrated circuit (DDIC). The CPU can be an application processor (AP). The CPU and GPU can be used to render and composite the image to be displayed on the display screen 130. The processor 110 may also include a neural network processing unit (NPU), a modem processor, etc.

[0170] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0171] The processor 110 may include a cache memory, which can be used to store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0172] The memory 120 may include a program storage area and a user data storage area. The program storage area may store the operating system and one or more applications (such as games), while the data storage area may store data created by the user during use of the terminal device 30 (such as photos and contacts). The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a hard disk, flash memory, or universal flash storage (UFS). The memory 120 may also be an external memory card, such as a Micro SD card.

[0173] The memory 120 may also store the implementation program of the reasoning operation method provided in the embodiments of this application on the terminal device. When the processor 110 reads and runs the program from the memory 120, the terminal device 30 can execute the method.

[0174] The memory 120 can also be integrated with the processor 110 into the integrated circuit of the SOC.

[0175] Terminal device 30 can realize display function through SOC, DDIC 140, and display screen 130.

[0176] The display screen 130 may include a display panel, a timing controller (TCON), etc. The display panel may include multiple pixel units, which may be an organic light-emitting diode (OLED) display panel, with each pixel unit including an OLED. The TCON is mainly used to connect the GPU or SOC to the display panel. After receiving image data or control signals from the GPU or SOC, it sets the timing of the control signals and data signals according to the relevant data or signals, and transmits the control signals and data signals to the display screen driver circuit according to the timing sequence, thereby driving the display panel to perform graphic display.

[0177] The display driver integrated circuit (DDIC) 140 serves as the control core of the display screen 130, driving the display screen 130 to operate and receiving data from the SOC (processor 110), such as image data and some instructions. The DDIC 140 can send drive signals and data to the display panel of the display screen 130 in the form of electrical signals, thereby controlling the screen brightness and color, enabling image information such as letters and pictures to be displayed on the screen and completing the screen refresh.

[0178] The image data to be displayed sent by the SOC to the DDIC 140 can be stored in the frame buffer to complete the display sending (or image sending). Then, the DDIC 140 retrieves the image data from the frame buffer and drives the display screen 130 to display it.

[0179] The wireless communication function of the terminal device 30 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0180] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 30 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0181] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 30. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0182] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 130. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0183] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 30, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0184] In some embodiments, antenna 1 of terminal device 30 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 30 to communicate with networks and other devices via wireless communication technology. Wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0185] The terminal device 30 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 130, and application processor.

[0186] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0187] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 30 may include one or N cameras 193, where N is a positive integer greater than 1.

[0188] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the terminal device 30 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0189] Video codecs are used to compress or decompress digital video. Terminal device 30 may support one or more video codecs. Thus, terminal device 30 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0190] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0191] Terminal device 30 can implement audio functions such as music playback and recording through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0192] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0193] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 30 can listen to music or make hands-free calls through the speaker 170A.

[0194] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 30 receives a phone call or voice message, the receiver 170B can be brought close to the listener's ear to receive the voice message.

[0195] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 30 may be equipped with at least one microphone 170C. In some embodiments, terminal device 30 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 30 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0196] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0197] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Terminal device 30 can receive button input and generate key signal inputs related to user settings and function control of terminal device 30. Motor 191 can generate vibration alerts. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with terminal device 30.

[0198] Figure 5 The illustrated structure does not constitute a specific limitation on terminal device 30. Terminal device 30 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

[0199] Figure 6 The server 20 provided in an embodiment of this application is shown.

[0200] Server 20 can be the server 20 mentioned in the foregoing embodiments. For example... Figure 6 As shown, server 20 may include: processor 210, memory 220, input / output device 230, communication module 240, etc., and these components can be coupled via a bus.

[0201] Server 20 may have powerful computing resources, and its processor 210 may include multiple processors, such as central processing unit (CPU), neural network processing unit (NPU), graphics processing unit (GPU), etc.

[0202] The processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0203] The processor 210 may have a cache memory, which can be used to store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the cache memory, which can reduce the waiting time of the processor 210 and improve the program running efficiency.

[0204] The processor 210 can also connect to external memory. This memory can be high-speed random access memory or non-volatile memory, such as a hard disk, flash memory, universal flash memory (UFS), etc. The memory can also be an external memory card, such as a Micro SD card.

[0205] The processor 210 is the computing core of the server 20, possessing powerful computing capabilities. Coupled with the memory 220, it can read and execute computer-readable instructions stored in the memory 220, running the operating system and various programs. Specifically, the CPU 210 can call programs stored in the memory 220, such as the implementation program of the inference calculation method provided in this embodiment on the server, and execute the instructions contained in that program.

[0206] The memory 220 may include high-speed random access memory, non-volatile memory, such as disk, flash memory, or other non-volatile solid-state storage devices. The memory 220 can be used to store various software programs and multiple sets of instructions. The memory 220 can store an operating system, such as Linux. The memory 220 can also store one or more programs, such as programs involved in patch creation, such as compilers and linkers. The memory 220 can also store the implementation program of the edge-cloud collaborative computing power scheduling method provided in the embodiments of this application.

[0207] Input / output device 230 may include devices such as a display screen, keyboard, and mouse, and can be used to receive user input and output program execution results to the user.

[0208] The communication module 240 may include a wired communication module and a wireless communication module. The wired communication module supports wired communication protocols such as Universal Serial Bus (USB), serial port, and Ethernet, communicating with other devices via physical communication cables. The wireless communication module may include 2G / 3G / 4G / 5G wireless communication modules, Wi-Fi communication modules, etc. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the CPU 210. The wireless communication module can also receive signals to be transmitted from the CPU 210, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0209] Figure 5 The illustrated structure does not constitute a limitation on server 20. Server 20 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

[0210] Figure 7 An electronic device 300 according to an embodiment of this application is illustrated. The electronic device 300 can be used to perform inference operations on the input of a first artificial intelligence program based on the aforementioned fusion large model, obtain inference results, and provide artificial intelligence services to the user. The electronic device 300 can be used to execute... Figure 4 Some or all of the steps in the method shown.

[0211] like Figure 7 As shown, the electronic device 300 may include: a program execution module 301, a loading module 303, and a reasoning and calculation module 305. Among them,

[0212] The program execution module 301 can be used to run the first artificial intelligence program.

[0213] The loading module 303 can be used to read the first model file from the first storage space and load the invertible matrix H and the quantized first artificial intelligence model stored in the first model file into memory.

[0214] The inference module 305 can be used to perform inference operations on the input of the first artificial intelligence program by running the quantized first artificial intelligence model. When performing inference operations, the input of the matrix multiplication operation is transformed by the invertible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0215] The first artificial intelligence model is the aforementioned fusion model, and the second artificial intelligence model is the aforementioned pre-fusion model. The first artificial intelligence model is obtained by fusing the invertible matrix H into the second artificial intelligence model. For details regarding the implementation of the steps by the program execution module 301, the loading module 303, and the inference operation module 305, please refer to the aforementioned method embodiments; they will not be repeated here.

[0216] Furthermore, the electronic device 300 can also have strong computing power and provide functions such as model construction and model quantization.

[0217] The electronic device 300 may further include: a model fusion module, which can be used to fuse the invertible matrix H into a second artificial intelligence model to obtain a first artificial intelligence model; the first artificial intelligence program provides artificial intelligence services to the user based on the second artificial intelligence model. Here, fusion can refer to: using the invertible matrix H to transform the weights of matrix multiplication operations in the second artificial intelligence model.

[0218] The electronic device 300 may further include a model quantization module and a storage module. The model quantization module is used to quantize the first artificial intelligence model to obtain a quantized first artificial intelligence model; the storage module is used to save the invertible matrix H and the quantized first artificial intelligence model to a first model file.

[0219] As mentioned earlier, the second artificial intelligence model may include L layers, where L is a positive integer and L > 2; the invertible matrix H includes L invertible matrices, and each of the L invertible matrices belongs to one of the L layers.

[0220] The model fusion module can be specifically used to transform the weights of matrix multiplication operations in each layer by utilizing the invertible matrix H of each layer in a hierarchical and parallel manner, thereby improving the efficiency of model fusion.

[0221] The model quantization module is specifically used to quantize the transformed weights of matrix multiplication operations in each layer, obtaining the quantized weights of matrix multiplication operations in each layer. This improves the efficiency of model quantization. The quantized weights of the first artificial intelligence model can include the quantized weights of matrix multiplication operations in each layer.

[0222] The electronic device 300 may also include: a learning module, which can be used to learn an invertible matrix H for integration into a first artificial intelligence model.

[0223] The learning module can be specifically used to learn the invertible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function using gradient descent. i Regarding the objective function and the application of gradient descent, please refer to the aforementioned method implementation examples, which will not be repeated here.

[0224] To improve learning efficiency, the learning module can be specifically used to learn the invertible matrices H of each layer of the first artificial intelligence model using different computing devices. Here, the computing devices may include graphics processing units (GPUs).

[0225] The specific implementation of each module included in the electronic device 300 can be found in the aforementioned embodiments, and will not be described in detail here.

[0226] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a transceiver or relay device. Alternatively, the processor and storage medium can exist as discrete components in a wireless access network device or user equipment.

[0227] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0228] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A method of inference computation, characterized by, The method comprises the following steps: running a first artificial intelligence program; reading a first model file from a first storage space, and loading a reversible matrix H and a quantized first artificial intelligence model saved in the first model file into a memory; the first artificial intelligence model is obtained by fusing the reversible matrix H into a second artificial intelligence model; the first artificial intelligence program provides an artificial intelligence service to a user based on the second artificial intelligence model; performing inference operation on an input of the first artificial intelligence program by using the quantized first artificial intelligence model loaded into the memory; when performing the inference operation, the input of a matrix multiplication operation in the quantized first artificial intelligence model is transformed by using the reversible matrix H in advance.

2. The method of claim 1, wherein, The method further comprises the following steps: fusing the reversible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model; wherein the fusing comprises transforming weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H.

3. The method of claim 2, wherein, The second artificial intelligence model comprises L layers, L is a positive integer, and L>2; the reversible matrix H comprises L reversible matrices, and the L reversible matrices belong to the L layers respectively; transforming the weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H, specifically comprising: transforming the weights of the matrix multiplication operation in each layer by using the reversible matrix H of each layer in a hierarchical and parallel manner.

4. The method according to any one of claims 2-3, characterized in that, Before running the first artificial intelligence program, the method further comprises the following steps: quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model; saving the reversible matrix H and the quantized first artificial intelligence model into the first model file.

5. The method of claim 4, wherein, Quantizing the first artificial intelligence model, specifically comprising: quantizing the transformed weights of the matrix multiplication operation in each layer to obtain quantized weights of the matrix multiplication operation in each layer; the quantized weights of the first artificial intelligence model comprise the quantized weights of the matrix multiplication operation in each layer.

6. The method according to any one of claims 2 to 5, characterized in that, The reversible matrix H for fusing into the second artificial intelligence model is obtained by learning.

7. The method of claim 6, wherein, The learning specifically comprises: The reversible matrix H of the i-th layer of the first artificial intelligence model is learned by minimizing the following objective function by the gradient descent method i : wherein f i represents an output of an i-th layer of the first artificial intelligence model before fusion, represents an output of an i-th layer of the first artificial intelligence model after fusion, L i represents an output and the output f i distance.

8. The method of claim 6 or 7, characterized in that, The learning specifically comprises: obtaining the reversible matrices of each layer of the first artificial intelligence model by different computing devices respectively; the computing devices comprise a graphics processing unit (GPU).

9. The method according to any one of claims 1 to 8, characterized in that, The reversible matrix H=H1*H2, H1 and H2 are also reversible matrices, and "*" represents a kronecker product of two matrices.

10. An electronic device, comprising: The method comprises the following steps: a program running module, configured to run a first artificial intelligence program; a loading module, configured to read a first model file from a first storage space, and load a reversible matrix H and a quantized first artificial intelligence model saved in the first model file into a memory; the first artificial intelligence model is obtained by fusing the reversible matrix H into a second artificial intelligence model; the first artificial intelligence program provides an artificial intelligence service to a user based on the second artificial intelligence model; an inference operation module, configured to perform inference operation on an input of the first artificial intelligence program by using the quantized first artificial intelligence model loaded into the memory; when performing the inference operation, the input of a matrix multiplication operation in the quantized first artificial intelligence model is transformed by using the reversible matrix H in advance.

11. The electronic device of claim 10, wherein, The method further comprises the following steps: The model fusion module is configured to fuse the invertible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model. The fusion includes transforming the weights of matrix multiplication in the second artificial intelligence model by using the invertible matrix H.

12. The electronic device of claim 11, wherein, Further comprising: The model quantization module is configured to quantize the first artificial intelligence model to obtain a quantized first artificial intelligence model. The saving module is configured to save the invertible matrix H and the quantized first artificial intelligence model into the first model file.

13. The electronic device of claim 12, wherein, The second artificial intelligence model includes L layers, L is a positive integer, and L>2. The invertible matrix H includes L invertible matrices, and the L invertible matrices belong to the L layers respectively. The model fusion module is specifically configured to transform the weights of matrix multiplication in each layer by using the invertible matrix H of each layer in a hierarchical and parallel manner.

14. The electronic device of claim 13, wherein, The model quantization module is specifically configured to quantize the transformed weights of matrix multiplication in each layer to obtain quantized weights of matrix multiplication in each layer. The quantized weights of the first artificial intelligence model include the quantized weights of matrix multiplication in each layer.

15. The electronic device of any of claims 10-14, wherein, Further comprising: The learning module is configured to learn the invertible matrix H for being fused into the second artificial intelligence model.

16. The electronic device of claim 15, wherein, The learning module is specifically configured to learn the reversible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function through the gradient descent method i : wherein f i represents an output of an i-th layer of the first artificial intelligence model before fusion, represents an output of an i-th layer of the first artificial intelligence model after fusion, L i represents an output and the output f i distance.

17. The electronic device of claim 15 or 16, wherein, The learning module is specifically configured to learn the invertible matrix of each layer of the first artificial intelligence model by different computing devices respectively. The computing devices include a graphics processing unit (GPU).

18. The electronic device of any of claims 10-17, wherein, The invertible matrix H=H1*H2, H1 and H2 are also invertible matrices, and "*" represents the kronecker product of two matrices.

19. An electronic device, comprising: The computer program is executed by the processor to implement the method of any one of claims 1-9. The computer program is executed by the processor to implement the method of any one of claims 1-9.

20. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-9.

21. A computer program product comprising a computer program, characterized in that, ​

Images