Inference operation method, and related device and system

Abstract

Embodiments of the present application provide an inference operation method, and a related device and system. In the embodiments of the present application, when a large model is quantized, a learned invertible matrix H is inserted at a matrix multiplication operation position in the large model, so as to suppress outliers in activation, and the quantization difficulty of activation is overcome by using an invertible matrix having a good effect of smoothing out outliers. Moreover, the input and weight of the matrix multiplication operation are both transformed once using the learned invertible matrix, so as to ensure the invariance of calculation. In the embodiments of the present application, the invertible matrix fused into the large model is learnable, and can adapt to the quantization of the large model, minimizing errors before and after model quantization, which can increase the accuracy of the model after model quantization.

Description

Inference operation method and related device and system

[0001] The present application claims priority to the Chinese patent application No. 202411266367.8, filed on September 10, 2024, with the State Intellectual Property Office of China, and entitled "Inference operation method and related device and system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of computers, and in particular to an inference operation method and related device and system. BACKGROUND

[0003] Many large language models (LLM) represented by ChatGPT have made remarkable achievements in the field of natural language processing (NLP). These large models have made breakthrough progress in various tasks such as language translation, text summarization, and question answering. However, these models usually have tens of billions or even hundreds of billions of parameters, requiring a large amount of storage and computing resources, and the actual deployment cost is high, making it difficult to deploy on resource-constrained devices such as mobile phones. SUMMARY

[0004] In a first aspect, the embodiments of the present application provide an inference operation method, which can perform inference operation on the input of an artificial intelligence program based on a fused large model after quantization, to obtain an inference result, which can effectively guarantee the accuracy of the inference operation.

[0005] The method can include: running a first artificial intelligence program; reading a first model file from a first storage space, and loading a reversible matrix H and a quantized first artificial intelligence model saved in the first model file into a memory; performing inference operation on the input of the first artificial intelligence program by the quantized first artificial intelligence model loaded into the memory, wherein when performing the inference operation, the input of the matrix multiplication operation is transformed by using the reversible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0006] In the first aspect, the first artificial intelligence model can be a fused large model in subsequent embodiments, which can be obtained by fusing the reversible matrix H into a second artificial intelligence model. The second artificial intelligence model can be a large model before fusion mentioned in subsequent embodiments. The first artificial intelligence program provides an artificial intelligence service based on the first artificial intelligence program. The reversible matrix H in the first model file can be used to reversibly transform the input of the matrix multiplication part in the first artificial intelligence model to ensure the invariance of the calculation.

[0007] The method provided in the first aspect can be executed by a terminal device such as a mobile phone or a tablet computer. The quantized first artificial intelligence model not only has a small model parameter quantity, but also has high precision after quantization due to the fusion of the reversible matrix, is suitable for deployment on a terminal device with limited resources, and can provide high inference operation accuracy. Of course, the method provided in the first aspect can also be executed by a cloud-side server, and can also reduce cloud-side resource occupation while ensuring model accuracy.

[0008] In the first aspect, before running the first artificial intelligence program, the first artificial intelligence model can also be constructed by fusing the reversible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model. The first artificial intelligence program can provide an artificial intelligence service to a user based on the second artificial intelligence model. The fusion can include transforming the weights of matrix multiplication in the second artificial intelligence model using the reversible matrix H. In this way, when quantizing the large model, the learned reversible matrix H is inserted at the position of matrix multiplication in the large model to suppress abnormal values in the activation and overcome the difficulty of quantizing the activation.

[0009] In combination with the first aspect, in some embodiments, the second artificial intelligence model can include L layers, L is a positive integer, and L>2. The reversible matrix H can include L reversible matrices, and the L reversible matrices belong to the L layers, respectively. Transforming the weights of matrix multiplication in the second artificial intelligence model using the reversible matrix H can specifically include transforming the weights of matrix multiplication in each layer using the reversible matrix H of each layer in a hierarchical and parallel manner. In this way, the model fusion efficiency can be improved by performing hierarchical and parallel model fusion.

[0010] The above step of constructing the first artificial intelligence model can be executed by a cloud-side server. After the first artificial intelligence model is constructed, the cloud-side server can send the model parameters (including weights) of the first artificial intelligence model to the terminal device, thereby reducing the requirement for the computing power of the terminal device. For a terminal device with strong computing power, this step can also be executed by the terminal device.

[0011] In combination with the first aspect, in some embodiments, before running the first artificial intelligence program, the method can further include quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model, and then saving the reversible matrix H and the quantized first artificial intelligence model into a first model file.

[0012] Quantizing the first artificial intelligence model can specifically include quantizing the transformed weights of matrix multiplication in each layer to obtain quantized weights of matrix multiplication in each layer. The quantized weights of the first artificial intelligence model can include the quantized weights of matrix multiplication in each layer. In this way, the model quantization efficiency can be improved by performing quantization in a multi-layer parallel manner.

[0013] The above step of quantizing the first artificial intelligence model has a large amount of calculation, can be executed by the cloud side, or can be executed by a terminal device with strong computing capability, and embodiments of the present application do not limit this.

[0014] In combination with the first aspect, in some embodiments, the reversible matrix H fused into the second artificial intelligence model is obtained by learning. In this way, the reversible matrix fused into the second artificial intelligence model is learnable, can adapt to large model quantization, realize error minimization before and after model quantization, and can improve the model precision after model quantization.

[0015] The specific implementation of learning can include:

[0016] The reversible matrix H of the i-th layer of the first artificial intelligence model is obtained by learning by minimizing the following objective function through the gradient descent method i :

[0017] Wherein, f i represents the output of the i-th layer of the first artificial intelligence model before fusion, represents the output of the i-th layer of the first artificial intelligence model after fusion, L i represents the distance between the output and the output f i .

[0018] In combination with the first aspect, in some embodiments, the reversible matrices of the respective layers of the first artificial intelligence model can be respectively obtained by learning through different computing devices, wherein the computing devices include a graphics processing unit (GPU). In this way, the learning speed of the reversible matrix can be improved, thereby improving the model compression efficiency, and the computing burden of a single GPU can be reduced, and the computing capability of a single GPU can only be able to bear the learning of a single layer of reversible matrix.

[0019] The above step of learning the reversible matrix can be executed by the cloud side, or can be executed by a terminal device with strong computing capability, and embodiments of the present application do not limit this.

[0020] In combination with the first aspect, in some embodiments, the large reversible matrix H can also be decomposed into the form of kronecker product of two smaller reversible matrices H1 and H2 to reduce the inference delay of the large model. That is, the reversible matrix H = H1 * H2, H1 and H2 are also reversible matrices, wherein "*" represents the kronecker product of two matrices.

[0021] In a second aspect, an embodiment of the present application provides a model quantization method, which can be applied to a cloud-side server. The method can include: the cloud-side server fusing a reversible matrix H into a second artificial intelligence model to obtain a first artificial intelligence model; the cloud-side server quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model; the cloud-side server saving the reversible matrix H and the quantized first artificial intelligence model into a first model file; and the cloud-side server sending the first model file to a terminal device.

[0022] Implementing the method of the second aspect can reduce the model parameter quantity on the terminal device side through quantization, so that a resource-constrained terminal device can also use a large model to perform inference operation; and when quantizing the large model, the abnormal values in the activation are suppressed by fusing the reversible matrix H into the large model, the model quantization difficulty is reduced, the model accuracy is effectively preserved, and thus the accuracy of the terminal device in performing inference operation using the quantized large model can be provided.

[0023] In combination with the second aspect, in some embodiments, fusing the reversible matrix H into the second artificial intelligence model can specifically include: the cloud-side server transforming the weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H.

[0024] In combination with the second aspect, in some embodiments, the second artificial intelligence model can include L layers, L being a positive integer and L>2. The reversible matrix H can include L reversible matrices, and the L reversible matrices respectively belong to the L layers. Transforming the weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H can specifically include: the cloud-side server layer-by-layer and in parallel transforming the weights of the matrix multiplication operation in each layer by using the reversible matrix H of each layer. In this way, the model fusion efficiency can be improved by layer-by-layer and parallel execution of model fusion.

[0025] In combination with the second aspect, in some embodiments, quantizing the first artificial intelligence model can specifically include: the cloud-side server quantizing the transformed weights of the matrix multiplication operation in each layer to obtain quantized weights of the matrix multiplication operation in each layer. The quantized weights of the first artificial intelligence model can include the quantized weights of the matrix multiplication operation in each layer. In this way, the model quantization efficiency can be improved by multi-layer and parallel execution of quantization.

[0026] In combination with the second aspect, in some embodiments, the reversible matrix H fused into the second artificial intelligence model is obtained through learning. In this way, the reversible matrix fused into the second artificial intelligence model is learnable, can adapt to large model quantization, realize error minimization before and after model quantization, and can improve the model accuracy after model quantization.

[0027] The specific implementation of learning can include:

[0028] The cloud-side server learns the reversible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function through gradient descent i :

[0029] where f i represents the output of the i-th layer of the first artificial intelligence model before fusion, represents the output of the i-th layer of the first artificial intelligence model after fusion, L i represents the distance between the output and the output f i .

[0030] In combination with the second aspect, in some embodiments, the cloud-side server can learn the reversible matrix of each layer of the first artificial intelligence model through different computing devices respectively, where the computing devices include a graphics processing unit (GPU). In this way, the cloud-side server can improve the learning speed of the reversible matrix, thereby improving the model compression efficiency, and can reduce the computing burden of a single GPU, and the computing capacity of a single GPU can only bear the learning of the reversible matrix of a single layer.

[0031] In combination with the second aspect, in some embodiments, the cloud-side server can also decompose the larger reversible matrix H into the form of the kronecker product of two smaller reversible matrices H1 and H2, to reduce the inference delay of the large model. That is, the reversible matrix H = H1 * H2, H1 and H2 are also reversible matrices, where "*" represents the kronecker product of two matrices.

[0032] In the third aspect, the embodiments of the present application provide an inference operation method, which can be applied to a terminal device. The method can include: receiving a first model file sent by a cloud-side server; saving the first model file to a first storage space; running a first artificial intelligence program; reading the first model file from the first storage space, and loading the reversible matrix H and the quantized first artificial intelligence model saved in the first model file into a memory; performing inference operation on the input of the first artificial intelligence program through the quantized first artificial intelligence model loaded into the memory, where when performing the inference operation, the input of the matrix multiplication operation is transformed by using the reversible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0033] In the third aspect, the first artificial intelligence model can be obtained by fusing the reversible matrix H into a second artificial intelligence model. The first artificial intelligence program provides an artificial intelligence service based on the first artificial intelligence program. The reversible matrix H in the first model file can be used to reversibly transform the input of the matrix multiplication part in the first artificial intelligence model, to ensure the invariance of the calculation.

[0034] Implementing the third approach allows for reasoning operations on the input of artificial intelligence programs based on the quantified fusion model, yielding reasoning results and effectively ensuring the accuracy of the reasoning operations.

[0035] In conjunction with the third aspect, in some embodiments, the terminal device may run the first artificial intelligence program after detecting a user operation that runs the first artificial intelligence program. This user operation may be, for example, clicking the desktop icon of the first artificial intelligence program, or navigating to the first artificial intelligence program from another application. The embodiments of this application do not limit the implementation of this operation. Not limited to running the first artificial intelligence program in response to a user operation, the terminal device may also run the first artificial intelligence program based on internal events, such as activating a smart voice assistant when the remaining battery power is high to provide companionship and conversation. The embodiments of this application do not limit the conditions for launching the first artificial intelligence program.

[0036] In conjunction with the third aspect, in some embodiments, the first artificial intelligence program may also be a resident program on the terminal device that starts running from the moment the device is powered on.

[0037] Fourthly, embodiments of this application provide an electronic device that may include multiple functional modules that cooperate with each other to implement the method described in the first aspect or any possible implementation of the first aspect.

[0038] Fifthly, embodiments of this application provide an electronic device that may include: a memory and a processor, the memory being coupled to the processor, a computer program being stored in the memory, and the computer program being executed by the processor to implement the method described in the first aspect or any possible implementation of the first aspect.

[0039] Sixthly, embodiments of this application provide a server that may include multiple functional modules that cooperate with each other to implement the method described in the second aspect or any possible implementation of the second aspect.

[0040] In a seventh aspect, embodiments of this application provide a server that may include: a memory and a processor, the memory being coupled to the processor, a computer program being stored in the memory, and the computer program being executed by the processor to implement the method described in the second aspect or any possible implementation of the second aspect.

[0041] Eighthly, embodiments of this application provide a terminal device, the server of which may include multiple functional modules that cooperate with each other to implement the method described in the third aspect or any possible implementation of the third aspect.

[0042] In a ninth aspect, an embodiment of the present application provides a terminal device, which can include a memory and a processor, the memory is coupled to the processor, and the memory stores a computer program, and the computer program is executed by the processor to implement the method described in the third aspect or any possible implementation manner of the third aspect.

[0043] In a tenth aspect, the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run by a processor, the method described in the first aspect or any possible implementation manner of the first aspect, or the method described in the second aspect or any possible implementation manner of the second aspect, or the method described in the third aspect or any possible implementation manner of the third aspect is implemented.

[0044] In an eleventh aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is run by a processor, the method described in the first aspect or any possible implementation manner of the first aspect, or the method described in the second aspect or any possible implementation manner of the second aspect, or the method described in the third aspect or any possible implementation manner of the third aspect is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0046] FIG. 1 shows an end-cloud model deployment system 10 provided by an embodiment of the present application;

[0047] FIG. 2 shows the activated abnormal values in the existing simple quantization and the activated abnormal values in Quarot quantization in comparison;

[0048] FIG. 3 shows a brief structure of a large model in an embodiment of the present application;

[0049] FIG. 4 shows the overall flow of an inference operation method provided by an embodiment of the present application;

[0050] FIG. 5 shows a terminal device 30 provided by an embodiment of the present application;

[0051] FIG. 6 shows a server 20 provided by an embodiment of the present application;

[0052] FIG. 7 shows an electronic device 300 provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] The terms used in the following embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to be limiting to the present application.

[0054] FIG. 1 shows an end-to-cloud model deployment system 10 provided by an embodiment of the present application.

[0055] As shown in FIG. 1, the end-to-cloud model deployment system 10 can include a server 20 and a terminal device 30. A communication link can be established between the server 20 and the terminal device 30. Among them,

[0056] The server 20 can be configured to train a large model such as a large language model (LLM) using a data set. The trained large model can provide various artificial intelligent (AI) functions such as language translation, text summarization, question answering, etc. However, the trained large model usually has tens of billions or even hundreds of billions of parameters, requiring a large amount of storage and computing resources, and is difficult to deploy on resource-constrained devices such as mobile phones.

[0057] The server 20 can also be configured to compress the large language model (LLM) by pruning, quantization, semi-structured sparsity, etc. Among them, quantization reduces the numerical precision of model parameters and intermediate activations, such as converting from 32-bit floating-point numbers to 8-bit or 4-bit integers, to reduce the parameter storage requirements of the model and speed up the calculation, so that the deployment of the large model on the terminal device 30 such as a mobile phone becomes no longer difficult.

[0058] The server 20 can also be configured to transmit the quantized model parameters to the terminal device 30 after completing quantization. Generally, the quantized model parameters can include quantized weights, such as weights with lower numerical precision.

[0059] The terminal device 30 can be configured to obtain the quantized model parameters from the server 20, load the quantized model parameters, and then use the quantized model parameters to perform inference operations on user inputs such as search terms and text to be translated, to obtain inference results. Here, the terminal device 30 loading the quantized model parameters can mean loading the quantized model parameters such as quantized weights into the memory of the terminal device 30.

[0060] In an embodiment of the present application, the server 20 can use a quantization method to compress the large model to realize model miniaturization, and deploy the model on terminal devices 30 such as mobile phones, tablet computers, smart screen devices, virtual reality (VR) devices, augmented reality (AR) devices, etc., so as to provide high-quality text summarization, context intelligent reply, advanced grammar correction, etc. artificial intelligent functions or services to users on the terminal device 30.

[0061] The quantization can include quantization aware training (QAT), post-training quantization (PTQ), etc. Compared with quantization aware training (QAT), post-training quantization (PTQ) does not require retraining of the model, does not require modification of the model architecture, has no requirements for the data set and training environment, and is a highly efficient and practical model compression method. The core idea of post-training quantization is that after the training of a large model is completed, the weights and activations of the model are quantized through a small amount of calibration data or without data to significantly reduce the storage requirements and computational overhead of the large model. The quantization objects of post-training quantization (PTQ) can include weights and activations. Activations are more difficult to quantize than weights because the weight distribution is generally more uniform, while there are a large number of outliers in the activations. The existence of outliers greatly increases the difficulty of quantization, making it difficult for simple quantization strategies to simultaneously preserve outliers and non-outliers with the same quantization parameters after quantization, especially in the low-bit quantization scenario, ultimately resulting in poor model compression results.

[0062] Even though activations are more difficult to quantize than weights due to the presence of outliers, different tokens exhibit similar changes across their channels, and outliers usually appear on fixed channels. Based on this observation, the SmoothQuant quantization method is proposed, which migrates outliers in activations to weights through equivalent scaling between channels to balance the quantization difficulty of weights and activations. For the SmoothQuant method, please refer to the following document: Guangxuan Xiao, et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models", International Conference on Machine Learning, 2023, the entire contents of which are incorporated herein by reference.

[0063] OmniQuant quantization method uses learnable scaling and offset coefficients to smooth and suppress outliers in activations based on SmoothQuant, and proposes a quantization method to learn the optimal scaling and offset coefficients by training and a small amount of calibration data. For the OmniQuant method, please further refer to the following literature: Wenqi Shao, et al., "OMNIQUANT: OMNIDIRECTIONALLY CALIBRATED QUANTIZATION FOR LARGE LANGUAGE MODELS", The International Conference on Learning Representations (ICLR), 2024, which is hereby incorporated by reference in its entirety herein.

[0064] However, SmoothQuant and OmniQuant only balance the quantization difficulty of weights and the quantization difficulty of activations, and change the size of outliers through vector multiplication, but the overall quantization difficulty of large models is still great, especially in low-precision quantization scenarios such as 4-bit, which cannot effectively preserve the ability and accuracy of large models.

[0065] QuaRot quantization method proposes to suppress outliers by Hadamard transformation of model weights through random Hadamard matrix, so that activations are easier to quantize. Figure 2 shows a comparison of outliers in activations when simple quantization and QuaRot quantization. As shown in Figure 2, when existing simple quantization is implemented on large models, there are outliers in activations, which significantly deviate from other values, resulting in high quantization difficulty; and after QuaRot quantization, outliers in activations can be eliminated, and the quantization difficulty is low. However, the QuaRot quantization method uses the same Hadamard transformation for all matrix multiplications to smooth outliers, does not take advantage of the regularity of outliers, and cannot effectively preserve the model's ability and accuracy. For the QuaRot method, please further refer to the following literature: Saleh Ashkboos, et al., "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs", The International Conference on Learning Representations (ICLR), 2024, which is hereby incorporated by reference in its entirety herein.

[0066] In order to effectively retain the model accuracy and capability while suppressing the abnormal values in the activation, in the embodiments of the present application, the server 20 can first insert the learned reversible matrix at the matrix multiplication position in the large model to perform model fusion, and then quantize the fused large model. In this way, the learned reversible matrix H can adapt to different abnormal value occurrence rules, better suppress the abnormal values in the activation, reduce the quantization difficulty, and realize the minimization of the error before and after the model quantization, thereby retaining the capability and accuracy of the large model.

[0067] Table 1 below shows the influence of several quantization methods on the model perplexity. Among them, “LLaMA-2-7B” represents a large language model with seventy billion (7B) parameters, “WikiText-2 perplexity” represents the perplexity of the model on the “WikiText-2” test data set, and “C4 perplexity” represents the perplexity of the model on the “C4” test data set. “FP16” represents that the model is not quantized, and the model uses high-precision sixteen-bit floating-point numbers. “SmoothQuant” and “OminiQuant” respectively represent the aforementioned SmoothQuant quantization method and OmniQuant quantization method. “AQuant” represents the quantization method based on the learned reversible matrix provided in the embodiments of the present application. As can be seen from Table 1, in terms of “WikiText-2 perplexity”, the perplexity of the model when it is not quantized (FP16) is lower, which is 5.48, at this time the model has strong capability and can well perform language prediction; after the SmoothQuant quantization model is used, the perplexity of the model is significantly increased, which is 83.12, and the prediction capability of the model is significantly reduced; after the OminiQuant quantization model is used, the perplexity of the model is also relatively high, which is 14.26, and the model capability is not good; and after the “AQuant” quantization model is used, the perplexity of the model is 5.77, which is close to the perplexity of “FP16”, which indicates that the capability of the model is effectively retained. Similarly, in terms of “C4 perplexity”, compared with other abnormal value suppression quantization methods, the “AQuant” quantization method can make the perplexity of the model closer to the perplexity of “FP16”, and can better retain the model capability.

[0068] Table 1

[0069] Table 2 below shows the influence of several quantization methods on the model accuracy. Among them, "LLaMA3-8B" represents a large language model with eight billion (8B) parameters, and "ARC-Challenge", "ARC-Easy", "HellaSwag", "LAMBADA", "PIQA", "WinoGrande" represent several different question and answer (Q&A) test data sets. "FP16" represents that the model is not quantized, "QuaRot" represents the aforementioned QuaRot quantization method, and "AQuant" represents the learned reversible matrix-based quantization method provided by the embodiments of the present application. As can be seen from Table 2, in terms of performance on several different question and answer (Q&A) test data sets, the model accuracy after "AQuant" quantization is closer to the accuracy when the model is not quantized (FP16) than the model accuracy after "QuaRot" quantization, and the model accuracy is more sufficient to retain.

[0070] Table 2

[0071] In addition to the fused large model quantized weights, the quantized model parameters transmitted by the server 20 to the terminal device 30 can further include the reversible matrix H. After receiving the quantized model parameters, the terminal device 30 can first transform and quantize the input using the reversible matrix H at the matrix multiplication operation in the large model, and then perform matrix multiplication operation on the transformed and quantized input and the quantized weights transformed by the reversible matrix H, to realize the invariance of model calculation.

[0072] In the large model, the learning of the reversible matrix H is performed in layers, and different layers can learn their respective reversible matrices in parallel. The reversible matrices of different layers can be distributed to different computing devices (such as graphic process units (GPUs)) for parallel learning, which can improve the model compression efficiency. How to learn the reversible matrix H will be described in detail in subsequent embodiments, which will not be expanded here.

[0073] The embodiments of the present application provide an inference operation method, which can be applied to the end-to-cloud model deployment system 10 shown in the example of FIG. 1. The inference operation method can be implemented based on the quantized fused large model, and the accuracy and precision of the inference operation are effectively guaranteed due to the quantized fused large model.

[0074] In order to effectively preserve the model accuracy, the main idea of quantizing the model can include: finding a reversible matrix with better effect of smoothing abnormal values based on learning, inserting the learned reversible matrix H at the matrix multiplication position in the large model when quantizing the large model (i.e. implementing model fusion) to suppress abnormal values in the activation, reduce the difficulty of quantization, and use the reciprocal of the reversible matrix and its inverse matrix to ensure the invariance of the calculation. Since the reversible matrix is learned, it can adapt to the abnormal value occurrence law in large model quantization, minimize the error before and after model quantization, and preserve the model capability after model quantization. Details are as follows.

[0075] 1. A learned reversible matrix can be added to a large model

[0076] As shown in FIG. 2, the large model in the embodiment of the present application can be, for example, a Transformer model, which can be divided into multiple layers, such as the 1st layer, the 2nd layer, etc. The output of the previous layer in the large model can be used as the input of the next layer, for example, the output f1 of the 1st layer is the input X1 of the 2nd layer. Each layer in the large model can include the same sub-layers, one of which is a self-attention (SelfAttention) and the other of which is a feed forward network (FFN). In the same layer, the output of the self-attention can be used as the input of the feed forward network.

[0077] Without considering other sub-layers such as addition and normalization sub-layers (Add&Norm), the output f i of the i-th layer can be simply represented by the following formula: i f i i = FFN i (SelfAttention i (x i )) (One)

[0078] where x i represents the input of the i-th layer, SelfAttention i represents the operation performed by the self-attention of the i-th layer, FFN i (SelfAttention i (x i )) represents the operation result of the self-attention which can be used as the input of the feed forward network.

[0079] The self-attention of each layer can include four linear sub-layers: Query layer, Key layer, Value layer, and Output layer. SelfAttention i (x i ) can be represented by the following formula:

[0080] wherein, denote the weights of the Query layer, the Key layer, the Value layer, and the Output layer in the self-attention of the i-th layer, respectively; denote...

[0081] Regarding FFN i The algorithm expansion of the formula can refer to the following document: Ashish Vaswani*, et al., "Attention Is All You Need", Advances in neural information processing systems, 2017, which is incorporated herein by reference in its entirety. The algorithm process of a single layer of the large model can also be implemented in other forms, and the embodiments of the present application do not limit this.

[0082] The f i is the output of the i-th layer of the large model without quantization, and the calculation amount is large, resulting in low model inference efficiency.

[0083] Matrix multiplication is widely used in large models such as Transformers, especially in the linear layer of the large model. The linear layer is widely used in the self-attention and the feedforward neural network in each layer of the large model, and plays a role in adjusting the dimension and transforming the features, providing better representation ability and fitting ability for the model.

[0084] In the embodiments of the present application, the server 20 can quantize the matrix multiplication in the large model to implement large model quantization, to reduce the storage space required by the large model and improve the model inference efficiency; and the server 20 can also transform the input and the weight of the matrix multiplication when quantizing, which can eliminate the influence of abnormal values in the activation on the quantization, and ensure the accuracy of the quantized model.

[0085] There are positions where matrix multiplication occurs in the self-attention sublayer of the i-th layer, such as in the above formula (2) is matrix multiplication. The feedforward network sublayer of the i-th layer can also include matrix multiplication. The present application uses to denote the output of the i-th layer of the large model after quantized matrix multiplication.

[0086] Without considering other sublayers that the layers can also include, such as addition and normalization sublayers (Add&Norm), the server 20 can determine the output of the i-th layer of the large model after quantized matrix multiplication by the following simplified formula

[0087] Where, x i This represents the input of the i-th layer; This indicates the operation performed by the self-attention of the i-th layer. The matrix multiplication operation in the operation undergoes quantization, and during quantization, the inputs and weights involved in the matrix multiplication operation are transformed using invertible matrices. This indicates the operation performed by the feedforward network (FFN) at layer i. The matrix multiplication operation in the operation undergoes quantization, and during quantization, the inputs and weights involved in the matrix multiplication operation are transformed using invertible matrices.

[0088] Server 20 can be calculated using the following formula (IV).

[0089] Among them, H i Let H represent the invertible matrix of the i-th layer of the large model. The invertible matrix H is inserted into the linear sub-layers such as the Query layer, Key layer, and Value layer within the self-attention mechanism of the i-th layer. i The specifics can differ.

[0090] Comparing equation (IV) with equation (II), it can be seen that the matrix multiplication operation in the large model is quantized, and the inputs and weights involved in the matrix multiplication operation during quantization are all invertible matrices H. i A transformation was performed.

[0091] For example, in equation (ii) This matrix multiplication operation is a linear matrix multiplication operation performed by the Query layer in the self-attention process, which becomes the operation in equation (iv). Where, x i H i , They can be represented using the invertible matrix H respectively. i The input x involved in this matrix multiplication operation i Weight Perform one transformation on each, int(x) i H i ), This can represent quantizing the transformed input and weights separately, with `int` representing the quantization operation. Similarly, in formula (II)... The matrix multiplication operation can also be quantized after transforming the input and weights. This reduces the difficulty of quantizing large models due to activation quantization.

[0092] While reducing the difficulty of quantization for large models caused by activation quantization, the model quantization method provided in this application can also keep the computational results of large models unchanged, that is, maintain computational invariance. The following uses the smallest matrix multiplication unit Y = XW to illustrate this effect, where Y represents the output of the matrix multiplication operation, X represents the input of the matrix multiplication operation, and W represents the weight of the matrix multiplication operation.

[0093] A simple quantization strategy is applied to quantize matrix multiplication, and its output is labeled as follows: It can be represented as follows:

[0094] In this embodiment of the application, the formula (v) used for quantizing matrix multiplication can be converted into:

[0095] It can be seen that by utilizing the inverse property of an invertible matrix and its inverse matrix, and performing the learned transformation and its inverse transformation on the input X and weight W respectively, the computational invariance of matrix multiplication can be achieved.

[0096] First, each matrix multiplication operation in the large model can incorporate an invertible matrix. The matrix multiplication operation that incorporates an invertible matrix can be represented as follows: 1. The weights of the matrix multiplication operation undergo a learnable transformation to obtain the transformed weights, which can be seen in formula (VI) under WH. -1 2. The input to the matrix multiplication operation undergoes a learnable transformation to obtain the transformed input, as shown in equation (VI) XH. Then, the large model incorporating invertible matrices can be quantized, specifically including: the transformed input and weights of the matrix multiplication operation can be quantized.

[0097] 2. Layered learning yields the invertible matrices of each layer of the large model.

[0098] In this embodiment of the application, an invertible matrix H is added to each layer of the large model. i It can be learned to adapt to the patterns of outlier occurrences, thereby helping to improve model accuracy.

[0099] For the i-th layer of a large model, server 20 can learn the optimal invertible matrix H of the i-th layer by minimizing the following objective function using the gradient descent algorithm. i :

[0100] Among them, f i This represents the output of the i-th layer of the previous large model. L represents the output of the i-th layer of the fused large model. i express with f idistance between the i-th layer of the pre-fusion model and the i-th layer of the post-fusion model. The objective function represented by the above equation (seven) is used to find the invertible matrix H i .

[0101] f i The calculation formula of the above equation (one) to (four) can be referred to, which can be substituted into equation (seven) to expand the objective function.

[0102] The gradient descent method can be specifically performed by calculating the gradient of the invertible matrix H with respect to the objective function, and iteratively finding the optimal invertible matrix that minimizes the objective training function value. One iteration process of the gradient descent method can be represented by the following algorithm:

[0103] wherein H i(j-1) represents the invertible matrix of the i-th layer determined by the j-1-th iteration of the gradient descent algorithm; H ij represents the invertible matrix of the i-th layer determined by the j-th iteration of the gradient descent algorithm; γ is a coefficient greater than 0, which can be adjusted in each iteration; represents the derivative of the objective training function with respect to H i(j-1) .

[0104] The f i in the above objective function is the output of the i-th layer of the pre-fusion model, and the output of each layer of the pre-fusion model can be calculated in advance by using the calibration data as the model input.

[0105] The server 20 can learn the optimal invertible matrix H i of each layer of the large model in parallel to improve the model compression and deployment efficiency. The invertible matrix learning between each layer of the large model can be independent, and the invertible matrix learning of each layer does not depend on the learning result of the previous layer. The learning between layers can be performed in parallel. The server 20 can distribute the invertible matrix learning of each layer to multiple computing devices, such as multiple GPUs, to learn the invertible matrix of each layer of the large model by different computing devices. In this way, the learning speed of the invertible matrix can be improved, and the model compression efficiency can be improved, and the computing burden of a single GPU can be reduced, and the computing capacity of a single GPU can only be able to bear the invertible matrix learning of a single layer.

[0106] Further, the server 20 can also decompose the larger invertible matrix H into the form of kronecker product of two smaller invertible matrices H1 and H2, so as to reduce the inference delay of the large model.

[0107] Figure 4 illustrates the overall flow of the inference operation method provided in the embodiments of this application. The terminal device and server in Figure 4 can be the terminal device 30 and server 20 in the system 10 shown in Figure 1, respectively. The inference operation method provided in the embodiments of this application will be described below.

[0108] S101, the server determines the invertible matrix H to be fused into the large model.

[0109] In this embodiment, the invertible matrix fused into the large model is learned, which enables the invertible matrix H to minimize the output deviation of each layer before and after model quantization after fusion into the large model, thereby achieving adaptive quantization for the model.

[0110] Specifically, the server can learn in layers to determine the invertible matrices of each layer that are integrated into the large model. For details on how to learn the invertible matrices of each layer, please refer to the previous sections; these will not be repeated here.

[0111] Here, fusion can refer to transforming the weights of matrix multiplication operations in the large model using an invertible matrix H. The invertible matrix H of the i-th layer of the large model... i It can be used to transform the weights of matrix multiplication operations in the i-th layer.

[0112] Invertible matrix H i This can be used to minimize the output deviation of the i-th layer before and after quantization. For the i-th layer of a large model, the server can train the invertible matrix H of the i-th layer using the aforementioned objective function. i Specifically: First, the server can input the text from a small amount of calibration data into the large model. During training, this input text can be used as the input X1 of the first layer, resulting in the unquantized matrix output f of each layer of the large model. i The output f of the i-th layer i The input x of the (i+1)th layer i+1 Then, the server can perform unquantized matrix multiplication on the output f of each layer. i Weight W i (such as the weights of the four linear layers in self-attention) The data is distributed to multiple computing devices, such as multiple GPUs, and these devices are controlled to work in parallel to obtain the invertible matrix H of each layer through the aforementioned objective function. i .

[0113] Calculate the output f of each layer before quantization matrix operation. i The formulas can be found in formulas (I) and (II) mentioned above, and will not be repeated here. Calculate the output of each layer after quantization matrix operations. The formulas can be found in formulas (III) and (IV) above, and will not be repeated here.

[0114] The hierarchical and parallel reversible matrix searching process for minimizing the error before and after quantization through the foregoing objective function can significantly improve the model compression efficiency and reduce the requirement for the operation capacity of a single operation device.

[0115] The calibration data can include multiple text inputs, which can be used as training data for multiple training to obtain a more adaptive reversible matrix.

[0116] In S102, the server fuses the reversible matrix H with the weight W of the large model, that is, fuses the reversible matrix H into the large model to obtain a fused large model.

[0117] The fusion can refer to transforming the weight of the matrix multiplication operation in the large model by using the reversible matrix H. Specifically, S102 can be implemented as follows: the server transforms the weight W of the matrix multiplication operation part in the large model by using the reversible matrix H to obtain the transformed weight, that is, the weight of the matrix multiplication operation in the fused large model, which can be referred to as WH in the foregoing formula (six). The weight of the matrix multiplication operation in the fused large model is transformed once compared with the weight of the matrix multiplication operation in the large model before fusion.

[0118] Suppose the large model includes L layers, L is a positive integer, and L>2. Then the reversible matrix fused into the large model can include L reversible matrices: H0, H1,..., HL-1, and HL. L The L reversible matrices belong to the L layers respectively. To improve the fusion efficiency, S102 can be executed hierarchically and in parallel in the large model. That is, the reversible matrix of each layer is used to transform the weight of the matrix multiplication operation in the layer hierarchically and in parallel. For the i-th layer of the large model, the server can use the reversible matrix H i of the i-th layer to transform the weight W i of the matrix multiplication operation part in the i-th layer, that is, multiply the weight W i by the reversible matrix H i to realize the reversible matrix H i fusion into the i-th layer of the large model.

[0119] In S103, the server quantizes the fused large model.

[0120] The server quantizing the fused large model can include: the server quantizing the weight of the matrix multiplication operation in the fused large model to obtain the quantized weight of the matrix multiplication operation, which can be referred to as int(WH -1 ) in the foregoing formula (six). The weight of the quantized fused large model (or the quantized weight of the fused large model) can include the quantized weight of the matrix multiplication operation in the fused large model.

[0121] S103In the large model can also be hierarchical at the same time, in order to improve the efficiency of quantization. That is, the server can be parallel quantization fusion large model in each layer of the matrix multiplication of the weight, get each layer of the matrix multiplication of the quantized weight, such as int(W1H1 -1 ), int(W2H2 -1 ),..., int(W L H L -1 ). Int(W1H1 -1 ), int(W2H2 -1 ,..., int(W L H L -1 ) respectively represent the first layer, the second layer,..., the L layer of the matrix multiplication of the fusion large model of the quantized weight.

[0122] The matrix multiplication in a certain layer using the quantization method provided in the application embodiment can be one or more matrix multiplications in the layer, or all matrix multiplications. The quantization method provided in the application embodiment can also only occur in part of the layers of the large model, and is not forced to occur in all layers of the large model.

[0123] The above S101-S103 can be used to realize the fusion of the large model and the model quantization, and prepare for the subsequent inference operation of the model.

[0124] S104, the server transmits the first model file to the terminal device, and the first model file includes the reversible matrix H and the quantized fusion large model. Correspondingly, the terminal device can receive the first model file.

[0125] The quantized fusion large model can include: the quantized weight of the fusion large model, specifically including the quantized weight of each layer of the fusion large model. The quantized weight of each layer can include the quantized weight of the matrix multiplication in the layer, such as the quantized weight of the first layer can include int(W1H1 -1 ), the quantized weight of the second layer can include int(W2H2 -1 ), int(W1H1 -1 ), int(W2H2 -1 ) respectively represent the first layer, the second layer of the matrix multiplication of the fusion large model of the quantized weight.

[0126] S105, the terminal device saves the first model file to the first storage space.

[0127] The server can also not directly transmit the invertible matrix H and the quantized fusion large model to the terminal device, but send a download address of the quantized fusion large model and a download address of the invertible matrix H to the terminal device. The terminal device can download the quantized fusion large model and the invertible matrix H from the cloud server hosting the fusion large model and the invertible matrix H according to the download addresses, and then save them into the first model file.

[0128] S106, the terminal device runs the first artificial intelligence program.

[0129] The first artificial intelligence program can be a system application or a third-party application on the terminal device that can provide an artificial intelligence service, or a certain artificial intelligence service provided by a certain system application or third-party application.

[0130] The first artificial intelligence program can provide an artificial intelligence service based on the large model, and can obtain an inference result by running the first artificial intelligence model to perform inference operation on the input of the first artificial intelligence program.

[0131] The terminal device can run the first artificial intelligence program after detecting a user operation of running the first artificial intelligence program. The user operation can be, for example, a user operation of clicking a desktop icon of the first artificial intelligence program, or a user operation of jumping from another application to the first artificial intelligence program. The implementation form of the operation is not limited in the embodiments of the present application. The terminal device can also run the first artificial intelligence program based on internal events, for example, starting a smart voice assistant to realize accompanying chat when the remaining power is large. The condition for starting the first artificial intelligence program is not limited in the embodiments of the present application.

[0132] The first artificial intelligence program can also be a resident program on the terminal device, which starts running as soon as the terminal device is powered on.

[0133] S107, the terminal device reads the first model file from the first storage space, and loads the invertible matrix H and the quantized fusion large model saved in the first model file into the memory.

[0134] The terminal device loading the quantized fusion large model into the memory can include: the terminal device loading the quantized weights of the fusion large model (including the quantized weights of the matrix multiplication in the fusion large model) into the memory.

[0135] S108, the terminal device can receive a user input X through the first artificial intelligence program in , such as the text to be translated.

[0136] S109, in response to the user input X in , the terminal device can use the quantized fusion large model to process the user input Xin performing the inference operation, the input of the matrix multiplication operation in the fused large model after quantization is transformed by using the reversible matrix H before the matrix multiplication operation.

[0137] The post-fusion large model (i.e., the fused large model) and the pre-fusion large model can be respectively referred to as a first artificial intelligence model and a second artificial intelligence model.

[0138] Taking the Transformer model shown in FIG. 3 as an example, in the fused large model after quantization, the inference operation can be performed layer by layer as follows:

[0139] In the first layer of the fused large model:

[0140] S1-1, the terminal device can first transform the user input X1 of the first layer by using the reversible matrix H1 of the first layer, and then quantize the transformed user input to obtain int(x1H1). X1=X in .

[0141] S1-2, then, the terminal device calculates the output of the first layer based on the quantization model structure of the first layer of the fused large model and int(x1H1).

[0142] wherein, respectively represent the quantization weights of several matrix multiplication parts in the first layer of the fused large model, and the several matrix multiplication parts are located in the Query layer, the Key layer, the Value layer, and the Output layer in the self-attention of the first layer. The first layer can also include matrix multiplication distributed in the FFN, not limited to the several matrix multiplications in the self-attention.

[0143] In the second layer of the fused large model:

[0144] S2-1, the terminal device can first transform the user input X2 of the second layer by using the reversible matrix H2 of the second layer, and then quantize the transformed user input to obtain int(x2H2).

[0145] S2-2, then, the terminal device calculates the output of the second layer based on the quantization model structure of the second layer of the fused large model and int(x2H2).

[0146] wherein, These represent the quantization weights of several matrix multiplication operations in the second layer of the fusion model. These matrix multiplication operations are located in the Query, Key, Value, and Output layers of the self-attention mechanism in the second layer. The second layer is not limited to these matrix multiplication operations in the self-attention mechanism; it can also include matrix multiplication operations distributed throughout the FFN.

[0147] And so on, in the i-th layer of the fusion large model (2 < i < L, where L is the number of layers in the large model):

[0148] Si-1, the terminal device can first utilize the invertible matrix H of the i-th layer. i For the user input X at level i i Perform the transformation, then quantize the transformed user input to obtain int(x) i H i ). That is, the input of each layer is equal to the output of the layer above.

[0149] Si-2, then, the terminal device is based on the quantized model structure of the i-th layer of the fusion large model and int(x) i H i Calculate the output of the i-th layer.

[0150] in, These represent the quantization weights of several matrix multiplication operations in the i-th layer of the fusion model. These matrix multiplication operations are located in the Query, Key, Value, and Output layers of the self-attention mechanism in the i-th layer. The i-th layer is not limited to these matrix multiplication operations in the self-attention mechanism; it can also include matrix multiplication operations distributed throughout the FFN.

[0151] Based on the above reasoning and calculation process, it can be concluded that S108 may include: at the matrix multiplication operation positions in each layer of the fusion large model, the terminal device can first utilize the invertible matrix H of each layer. i Input x for matrix multiplication operations at each layer i The input is transformed, then quantized, and then the quantized transformed input (which can be represented as int(x)) is obtained through matrix multiplication operations in each layer. i H i The quantization weights of matrix multiplication operations in each layer (which can be represented as) like Perform matrix multiplication.

[0152] The flow shown in FIG. 4 can also be executed on a single device, for example, both on a server or on a terminal device with strong storage and computing capabilities. If executed on a single device, step S104 in the above flow can be omitted. The electronic device shown in the embodiment of FIG. 7 can be an implementation of the single device.

[0153] Not limited to large models, the embodiments of the present application can also be applicable to models with other numbers of parameters, and slightly smaller models can also use the method provided by the embodiments of the present application for quantization.

[0154] FIG. 5 shows a terminal device 30 provided by an embodiment of the present application.

[0155] The terminal device 30 can be the terminal device 30 mentioned in the foregoing embodiments. The device type of the terminal device 30 can be any one of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and a smart large screen, a smart speaker, and the like smart home device, a smart bracelet, a smart watch, smart glasses, and the like wearable device, an augmented reality (AR), a virtual reality (VR), a mixed reality (MR), and the like extended reality (XR) device, a vehicle-mounted device or a smart city device, and the like.

[0156] As shown in FIG. 5, the terminal device 30 can include a processor 110, a memory 120, a display screen 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, and the like. The various components in the terminal device 30 can be connected through a bus.

[0157] The processor 110 can be one or more, which can be integrated in an integrated circuit of a system on chip (SOC). The SOC is a system-level chip. The processor 110 can include a central processing unit (CPU), a graphic processing unit (GPU), a display driver integrated circuit (DDIC). The CPU can be an application processor (AP). The CPU and the GPU can be used to render a picture to be sent to the display 130. The processor 110 can further include a neural-network processing unit (NPU), a modem processor, and the like.

[0158] The processor 110 can include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, and the like.

[0159] The processor 110 can be provided with a cache memory, which can be used to save instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0160] The memory 120 can include a program storage area and a user data storage area. The program storage area can store an operating system and one or more application programs (e.g., a game application), and the data storage area can store data (e.g., photos, contacts) created by a user during use of the terminal device 30. The memory 120 can be a high-speed random access memory or a nonvolatile memory such as a magnetic disk, a flash memory, a universal flash storage (UFS), or the like. The memory 120 can also be an external memory card such as a Micro SD card.

[0161] The memory 120 can also store an implementation program of the inference operation method provided in the embodiments of the present application on the terminal device. When the processor 110 reads and runs the program from the memory 120, the terminal device 30 can execute the method.

[0162] The memory 120 can also be integrated with the processor 110 in an integrated circuit of a SOC.

[0163] The terminal device 30 can achieve a display function through a SOC, a DDIC 140, and a display screen 130.

[0164] The display screen 130 can include a display panel and a timing controller (TCON). The display panel can include a plurality of pixel units, which can be an organic light-emitting diode (OLED) display panel. Each pixel unit includes an OLED. The TCON is mainly used to connect a GPU or a SOC to the display panel. After receiving image data or control signals from the GPU or the SOC, the TCON sets the timing of the control signals and the data signals according to the relevant data or signals, and transmits the control signals and the data signals to the display screen driving circuit according to the timing, so as to achieve the purpose of driving the display panel to display graphics.

[0165] The display driving integrated circuit (DDIC) 140 can be used as a control core of the display screen 130, drive the display screen 130 to work, and receive data such as image data and some instructions from the SOC (processor 110). The DDIC 140 can send driving signals and data to the display panel of the display screen 130 in the form of electrical signals, so as to control the screen brightness and color, and make image information such as letters and pictures appear on the screen, and complete screen refresh.

[0166] The image data of the to-be-displayed picture sent by the SOC to the DDIC 140 can be stored in a frame buffer to complete display (or image sending). Then, the DDIC 140 takes out the image data from the frame buffer and drives the display screen 130 to display.

[0167] The wireless communication function of the terminal device 30 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0168] The antenna 1 and the antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the terminal device 30 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0169] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the terminal device 30. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert the signals into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be arranged in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be arranged in the same device.

[0170] The modem processor can include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the loudspeaker 170A and the microphone 170B, etc.), or displays an image or a video through the display screen 130. In some embodiments, the modem processor can be an independent device. In some other embodiments, the modem processor can be independent of the processor 110, and arranged in the same device as the mobile communication module 150 or other functional modules.

[0171] The wireless communication module 160 can provide a wireless communication solution including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the terminal device 30. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0172] In some embodiments, the antenna 1 and the mobile communication module 150 of the terminal device 30 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the terminal device 30 can communicate with the network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0173] The terminal device 30 can realize the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 130, and the application processor, etc.

[0174] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing, and the image visible to the naked eye is converted. The ISP can also optimize the algorithm of the noise and brightness of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0175] The camera 193 is used to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in a format such as RGB, YUV, etc. In some embodiments, the terminal device 30 can include one or N cameras 193, where N is a positive integer greater than 1.

[0176] The digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 30 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0177] The video codec is used to compress or decompress digital videos. The terminal device 30 can support one or more video codecs. In this way, the terminal device 30 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0178] The NPU is a neural-network (NN) computing processor, which is inspired by the structure of biological neural networks, such as the transmission mode between human brain neurons, and can quickly process input information and continuously self-learn. Through the NPU, the terminal device 30 can implement intelligent cognitive applications such as image recognition, face recognition, speech recognition, and text understanding.

[0179] The terminal device 30 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0180] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or part of the functions of the audio module 170 can be disposed in the processor 110.

[0181] The speaker 170A, also called a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The terminal device 30 can listen to music or listen to a hands-free call through the speaker 170A.

[0182] The receiver 170B, also called a "earpiece", is used to convert an audio electrical signal into a sound signal. When the terminal device 30 answers a call or a voice message, the user can listen to the voice by holding the receiver 170B close to the ear.

[0183] The microphone 170C, also called a "microphone", "sound pickup", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can make a sound by holding the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The terminal device 30 can be provided with at least one microphone 170C. In other embodiments, the terminal device 30 can be provided with two microphones 170C, in addition to collecting sound signals, it can also realize the function of noise reduction. In other embodiments, the terminal device 30 can also be provided with three, four or more microphones 170C, which can realize the functions of collecting sound signals, noise reduction, identifying sound sources, realizing directional recording, etc.

[0184] The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0185] The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key. It can also be a touch key. The terminal device 30 can receive a key input and generate a key signal input related to the user settings and function control of the terminal device 30. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the terminal device 30.

[0186] The structure shown in FIG. 5 does not constitute a specific limitation on the terminal device 30. The terminal device 30 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0187] FIG. 6 shows a server 20 provided by an embodiment of the present application.

[0188] The server 20 can be the server 20 mentioned in the foregoing embodiments. As shown in FIG. 6, the server 20 can include a processor 210, a memory 220, an input output device 230, a communication module 240, and the like, which can be coupled through a bus.

[0189] The server 20 can have powerful computing resources, and the processor 210 thereon can include multiple processors, which can be, for example, a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), or the like.

[0190] The processor 210 can include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, and the like.

[0191] The processor 210 can be provided with a cache memory, which can be used to save instructions or data that have just been used or recycled by the processor 210. If the processor 210 needs to use the instructions or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 210 and improve the program running efficiency.

[0192] The processor 210 can also be externally connected to a memory. The memory can be a high-speed random access memory, or a non-volatile memory such as a disk, a flash memory, a universal flash storage (UFS), or the like. The memory can also be an external memory card such as a Micro SD card.

[0193] The processor 210 is the computing core of the server 20 and has powerful computing capability. It is coupled with the memory 220 and can be used to read and execute computer readable instructions in the memory 220, run an operating system and various programs. Specifically, the CPU 210 can be used to call programs stored in the memory 220, such as an implementation program of the inference operation method provided in the embodiments of the present application on the server, and execute instructions contained in the program.

[0194] The memory 220 can include a high-speed random access memory, a non-volatile memory such as a magnetic disk, a flash memory or other non-volatile solid-state storage device. The memory 220 can be used to store various software programs, groups of instructions. The memory 220 can store an operating system, such as a Linux operating system or the like. The memory 220 can also store one or more programs, such as programs involved in patch making, such as a compiler, a linker. The memory 220 can also store an implementation program of the end-cloud collaborative computing power scheduling method provided by the embodiments of the present application.

[0195] The input and output device 230 can include a display screen, a keyboard, a mouse and the like, and can be used to receive user input and output program running results to the user.

[0196] The communication module 240 can include a wired communication module and a wireless communication module. The wired communication module can support wired communication protocols such as universal serial bus (USB), serial port, Ethernet and the like, and communicate with other devices through physical communication cables. The wireless communication module can include 2G / 3G / 4G / 5G and the like wireless communication module, Wi-Fi communication module and the like. The wireless communication module receives electromagnetic waves via an antenna, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the CPU 210; the wireless communication module can also receive signals to be sent from the CPU 210, frequency modulate them, amplify them, and radiate them as electromagnetic waves via an antenna.

[0197] The structure shown in FIG. 5 does not constitute a limitation on the server 20. The server 20 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0198] FIG. 7 shows an electronic device 300 provided by an embodiment of the present application. The electronic device 300 can be used to infer the input of the first artificial intelligence program based on the foregoing fusion large model, obtain an inference result, and provide an artificial intelligence service to a user. The electronic device 300 can be used to perform part or all of the steps in the method shown in FIG. 4.

[0199] As shown in FIG. 7, the electronic device 300 can include a program running module 301, a loading module 303 and an inference operation module 305. Wherein,

[0200] The program running module 301 can be used to run the first artificial intelligence program.

[0201] The loading module 303 can be used to read the first model file from the first storage space, and load the reversible matrix H and the quantized first artificial intelligence model saved in the first model file into the memory.

[0202] The inference operation module 305 can be configured to perform inference operation on the input of the first artificial intelligence program by running the quantized first artificial intelligence model, and when performing the inference operation, the input of the matrix multiplication operation is transformed by using the reversible matrix H before the matrix multiplication operation in the quantized first artificial intelligence model.

[0203] The first artificial intelligence model is the aforementioned fusion large model, and the second artificial intelligence model is the aforementioned large model before fusion. The reversible matrix H is fused into the second artificial intelligence model to obtain the first artificial intelligence model. For implementation details of the respective steps performed by the program running module 301, the loading module 303, and the inference operation module 305, refer to the aforementioned method embodiments, which will not be described here.

[0204] Further, the electronic device 300 can also have strong computing power and can provide functions such as model construction and model quantization.

[0205] The electronic device 300 can further include a model fusion module configured to fuse the reversible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model; and the first artificial intelligence program provides artificial intelligence services to users based on the second artificial intelligence model. The fusion can refer to transforming the weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H.

[0206] The electronic device 300 can further include a model quantization module and a saving module. The model quantization module can be configured to quantize the first artificial intelligence model to obtain a quantized first artificial intelligence model; and the saving module can be configured to save the reversible matrix H and the quantized first artificial intelligence model into a first model file.

[0207] As described above, the second artificial intelligence model can include L layers, L is a positive integer, and L>2; the reversible matrix H includes L reversible matrices, and the L reversible matrices belong to the L layers, respectively.

[0208] The model fusion module can be specifically configured to transform the weights of the matrix multiplication operation in each layer by using the reversible matrix H of each layer in a hierarchical and parallel manner to improve the model fusion efficiency.

[0209] The model quantization module can be specifically configured to quantize the transformed weights of the matrix multiplication operation in each layer to obtain quantized weights of the matrix multiplication operation in each layer. In this way, the model quantization efficiency can be improved. The quantized weights of the first artificial intelligence model can include the quantized weights of the matrix multiplication operation in each layer.

[0210] The electronic device 300 can further include a learning module configured to learn the reversible matrix H for being fused into the first artificial intelligence model.

[0211] The learning module can be specifically configured to learn the reversible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function through gradient descent method i For the objective function and the application of the gradient descent method, reference can be made to the foregoing method embodiments, which will not be described herein again.

[0212] To improve the learning efficiency, the learning module can be specifically configured to learn the reversible matrix H of each layer of the first artificial intelligence model through different computing devices respectively. Here, the computing devices can include a graphics processing unit (GPU).

[0213] For the specific implementation of each module included in the electronic device 300, reference can be made to the foregoing embodiments, which will not be described herein again.

[0214] The steps of the methods or algorithms described in connection with the disclosure of the embodiments disclosed herein can be implemented in hardware, or be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a RAM, a flash memory, a ROM, an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disk read-only memory (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a transceiver or a relay device. Of course, the processor and the storage medium can also exist as discrete components in a wireless access network device or a user equipment.

[0215] Those skilled in the art should be aware that, in one or more of the examples described above, the functions described in the embodiments disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. The computer-readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. The storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0216] The above detailed description of the embodiments of the present application is merely intended to provide a further detailed description of the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above is only a specific implementation of the embodiments of the present application, and is not used to limit the protection scope of the embodiments of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method of inference computation, characterized by, The method comprises the following steps: running a first artificial intelligence program; reading a first model file from a first storage space, and loading a reversible matrix H and a quantized first artificial intelligence model saved in the first model file into a memory; the first artificial intelligence model is obtained by fusing the reversible matrix H into a second artificial intelligence model; the first artificial intelligence program provides an artificial intelligence service to a user based on the second artificial intelligence model; performing inference operation on an input of the first artificial intelligence program by using the quantized first artificial intelligence model loaded into the memory; when performing the inference operation, the input of a matrix multiplication operation in the quantized first artificial intelligence model is transformed by using the reversible matrix H in advance.

2. The method of claim 1, wherein, The method further comprises the following steps: fusing the reversible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model; wherein the fusing comprises transforming weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H.

3. The method of claim 2, wherein, The second artificial intelligence model comprises L layers, L is a positive integer, and L>2; the reversible matrix H comprises L reversible matrices, and the L reversible matrices belong to the L layers respectively; transforming the weights of the matrix multiplication operation in the second artificial intelligence model by using the reversible matrix H, specifically comprising: transforming the weights of the matrix multiplication operation in each layer by using the reversible matrix H of each layer in a hierarchical and parallel manner.

4. The method according to any one of claims 2-3, characterized in that, Before running the first artificial intelligence program, the method further comprises the following steps: quantizing the first artificial intelligence model to obtain a quantized first artificial intelligence model; saving the reversible matrix H and the quantized first artificial intelligence model into the first model file.

5. The method of claim 4, wherein, Quantizing the first artificial intelligence model, specifically comprising: quantizing the transformed weights of the matrix multiplication operation in each layer to obtain quantized weights of the matrix multiplication operation in each layer; the quantized weights of the first artificial intelligence model comprise the quantized weights of the matrix multiplication operation in each layer.

6. The method according to any one of claims 2 to 5, characterized in that, The reversible matrix H for fusing into the second artificial intelligence model is obtained by learning.

7. The method of claim 6, wherein, The learning specifically comprises: The reversible matrix H of the i-th layer of the first artificial intelligence model is learned by minimizing the following objective function by the gradient descent method i : wherein f i denotes an output of the i-th layer of the first artificial intelligence model before fusion, represents the output of the i-th layer of the first artificial intelligence model after fusion, L i represents the output The distance of the output f i from the origin.

8. The method of claim 6 or 7, characterized in that, The learning specifically comprises: obtaining the reversible matrices of each layer of the first artificial intelligence model by different computing devices respectively; the computing devices comprise a graphics processing unit (GPU).

9. The method according to any one of claims 1 to 8, characterized in that, The reversible matrix H=H1*H2, H1 and H2 are also reversible matrices, and "*" represents a kronecker product of two matrices.

10. An electronic device, comprising: The method comprises the following steps: a program running module, configured to run a first artificial intelligence program; a loading module, configured to read a first model file from a first storage space, and load a reversible matrix H and a quantized first artificial intelligence model saved in the first model file into a memory; the first artificial intelligence model is obtained by fusing the reversible matrix H into a second artificial intelligence model; the first artificial intelligence program provides an artificial intelligence service to a user based on the second artificial intelligence model; an inference operation module, configured to perform inference operation on an input of the first artificial intelligence program by using the quantized first artificial intelligence model loaded into the memory; when performing the inference operation, the input of a matrix multiplication operation in the quantized first artificial intelligence model is transformed by using the reversible matrix H in advance.

11. The electronic device of claim 10, wherein, The method further comprises the following steps: The model fusion module is configured to fuse the invertible matrix H into the second artificial intelligence model to obtain the first artificial intelligence model. The fusion includes transforming the weights of matrix multiplication in the second artificial intelligence model by using the invertible matrix H.

12. The electronic device of claim 11, wherein, Further comprising: The model quantization module is configured to quantize the first artificial intelligence model to obtain a quantized first artificial intelligence model. The saving module is configured to save the invertible matrix H and the quantized first artificial intelligence model into the first model file.

13. The electronic device of claim 12, wherein, The second artificial intelligence model includes L layers, L is a positive integer, and L>2. The invertible matrix H includes L invertible matrices, and the L invertible matrices belong to the L layers respectively. The model fusion module is specifically configured to transform the weights of matrix multiplication in each layer by using the invertible matrix H of each layer in a hierarchical and parallel manner.

14. The electronic device of claim 13, wherein, The model quantization module is specifically configured to quantize the transformed weights of matrix multiplication in each layer to obtain quantized weights of matrix multiplication in each layer. The quantized weights of the first artificial intelligence model include the quantized weights of matrix multiplication in each layer.

15. The electronic device of any of claims 10-14, wherein, Further comprising: The learning module is configured to learn the invertible matrix H for being fused into the second artificial intelligence model.

16. The electronic device of claim 15, wherein, The learning module is specifically configured to learn the reversible matrix H of the i-th layer of the first artificial intelligence model by minimizing the following objective function through the gradient descent method i : wherein f i represents an output of the i-th layer of the first artificial intelligence model before fusion, represents the output of the i-th layer of the first artificial intelligence model after fusion, L i represents the output Distance to output f i of the distance.

17. The electronic device of claim 15 or 16, wherein, The learning module is specifically configured to learn the invertible matrix of each layer of the first artificial intelligence model by different computing devices respectively. The computing devices include a graphics processing unit (GPU).

18. The electronic device of any of claims 10-17, wherein, The invertible matrix H=H1*H2, H1 and H2 are also invertible matrices, and "*" represents the kronecker product of two matrices.

19. An electronic device, comprising: The computer program is executed by the processor to implement the method of any one of claims 1-9. The computer program is executed by the processor to implement the method of any one of claims 1-9.

20. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-9.

21. A computer program product comprising a computer program, characterized in that, ​

Images