Ai model inference method and apparatus

Abstract

Provided are an AI model inference method and apparatus. The method comprises: acquiring input data; and inputting the input data into an AI model to obtain an inference result, wherein the AI model comprises a plurality of feature extraction network layers, at least one feature extraction network layer among the plurality of feature extraction network layers comprises a first neural network layer and a feature dictionary layer, the first neural network layer is connected to the feature dictionary layer, and the feature dictionary layer is used for performing dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matching the first feature vector. In the AI model provided in the present application, a feature dictionary query operation replaces previous linear calculation that is computationally intensive, thereby significantly reducing computational overhead of the AI model during inference while ensuring model accuracy, implementing inference acceleration of the AI model, and reducing an output delay of the AI model.

Description

Inference method and device of AI model

[0001] The present application claims priority to the Chinese patent application No. 202410836122.8, filed on June 25, 2024, entitled "Inference method and device of AI model", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an inference method and device of an AI model. BACKGROUND

[0003] Large language models (LLMs) based on Transformers have been widely applied in general and special field tasks and have achieved great success. However, with the development of LLMs, the model parameters of LLMs increase rapidly (for example, the model parameters of LLMs reach hundreds of billions), and the computational overhead of the model inference process of LLMs is increasing, and the inference time is getting longer, resulting in an increase in the response delay of LLMs and a poor user experience. SUMMARY

[0004] Embodiments of the present application provide an inference method and device of an AI model, which replace the previous linear calculation that consumes a lot of computation by a feature dictionary query operation, so that the AI model significantly reduces the computational overhead in the inference process on the basis of ensuring the model accuracy, realizes the inference acceleration of the AI model, and reduces the output delay of the AI model.

[0005] In a first aspect, the present application provides an inference method of an AI model, comprising: obtaining input data, the input data comprising at least one of text data, voice data, image data and video data; inputting the input data into the AI model to obtain an inference result, the AI model comprising a plurality of feature extraction network layers, at least one of the plurality of feature extraction network layers comprising a first neural network layer and a feature dictionary layer, the first neural network layer being connected to the feature dictionary layer, the feature dictionary layer being configured to perform a dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matched with the first feature vector, the second feature vector being used as an output vector of the feature extraction network layer to which the first neural network layer belongs.

[0006] The inference method of the AI model provided in the application replaces the linear layer in the original AI model with a feature dictionary layer. Thus, in the inference process of the AI model, the matrix multiplication calculation of the original linear layer is replaced by a query operation of the feature dictionary, which significantly reduces the computational overhead of the inference process of the AI model, realizes the inference acceleration of the AI model, reduces the output delay of the AI model, and hardly reduces the accuracy of the AI model.

[0007] Taking a large language model based on a transformer architecture as an example, the feature extraction network in the large language model includes multiple encoders, which can be referred to as multiple feature extraction network layers. At least one encoder (for the sake of distinguishing from other encoders, referred to as a feature dictionary encoder) in the multiple encoders includes a self-attention layer (i.e., a first neural network layer) and a feature dictionary layer connected to the self-attention layer. Compared with the existing encoder including a self-attention layer and a feedforward neural network layer, the feature dictionary encoder replaces the feedforward neural network layer, and does not need to perform linear calculation which consumes more computing power, but instead performs a dictionary query operation which saves computing power, so as to obtain an output feature vector of the encoder.

[0008] It is easy to understand that the multiple encoders in the large language model perform feature extraction on input data (the input data is generally encoded into an embedding vector by an embedding layer before inputting into the frontmost encoder), and the feature vector output by the last encoder is the final feature vector. For example, the multiple encoders in the large language model include a first encoder, a second encoder, …, and an Nth encoder. The first encoder performs feature extraction on the embedding vector output by the embedding layer, the second encoder is used to update the feature vector output by the first encoder, and outputs an updated feature vector, and the Nth encoder is used to update the feature vector output by the Nth-1 encoder, and outputs the final feature vector. When the feature dictionary encoder is located in the last encoder (i.e., the Nth encoder), the feature dictionary encoder includes a self-attention layer and a feature dictionary layer connected to the self-attention layer. The output of the self-attention layer is the first feature vector, and the output of the feature dictionary layer is the second feature vector. The second feature vector is the output of the feature dictionary encoder, and the feature vector output by the feature dictionary encoder is the final feature vector.

[0009] In a possible implementation, the feature dictionary layer includes a first feature dictionary, the first feature dictionary includes K dictionary tables, each of the K dictionary tables includes a plurality of basis vectors, K is a positive integer; and the feature dictionary layer is configured to perform a dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matched with the first feature vector, and a specific implementation of the feature dictionary layer is: based on the first feature vector, obtaining an index value and a weight value of each of the K dictionary tables; based on the index value of each of the K dictionary tables, searching for a target basis vector corresponding to each of the K dictionary tables from each of the K dictionary tables; and based on the weight value of each of the K dictionary tables, performing a weighted summation on the target basis vector corresponding to each of the K dictionary tables to obtain the second feature vector.

[0010] The output of the original linear layer (for example, a feedforward neural network layer) is represented in the form of combination of a plurality of basis vectors (that is, weighted summation of a plurality of basis vectors), so that the computing power consumption in the AI model inference process is reduced, and the inference speed of the AI model is increased.

[0011] In a possible implementation, the feature dictionary layer further includes a query sub-layer; and a specific implementation of obtaining the index value and the weight value of each of the K dictionary tables based on the output vector is: taking the output vector as an input of the query sub-layer, and outputting the index value and the weight value of each of the K dictionary tables.

[0012] Optionally, the plurality of basis vectors in each of the dictionary tables and the query sub-layer are trained based on a training sample set. That is, the input and output information of the linear layer in the original AI model can be saved in the form of learnable parameters, and can be learned by dictionary learning. The output of the original linear layer (for example, a linear layer in a feedforward neural network or a self-attention layer) is represented in the form of combination of basis vectors, so that, in the AI model inference process, linear calculation with high computing power consumption is not needed, but dictionary query operation with low computing power consumption is performed, so that the computing power consumption is effectively reduced, and the inference speed of the AI model is increased.

[0013] In a possible implementation, the feature dictionary layer includes a second feature dictionary, the second feature dictionary is constructed based on a plurality of input feature and output feature pairs, the plurality of input feature and output feature pairs are obtained by collecting input features and output features of the linear layer after the training sample set is input into the AI model before optimization, and at least one feature extraction network layer of the AI model before optimization includes the first neural network layer and the linear layer.

[0014] It needs to be explained that the AI model before optimization here is relative to the AI model of the application, and the AI model before optimization and the AI model of the application belong to the same type of AI model. The difference is that the AI model before optimization is the structure of the existing AI model. For example, the AI model before optimization and the AI model of the application are both large language models based on transformer architecture. At least one feature extraction network layer of the AI model before optimization includes a first neural network layer and a linear layer, while the corresponding feature extraction network layer in the AI model of the application includes a first neural network layer and a feature dictionary layer. That is, the application replaces the linear layer in the previous AI model with the feature dictionary layer. Therefore, in the AI model inference process, there is no need to perform linear calculation which consumes a lot of computing power, but instead, dictionary query operation which saves computing power is performed, effectively reducing the computing power overhead and increasing the inference speed of the AI model.

[0015] In one possible implementation, the second feature dictionary includes a first lookup tree, the first lookup tree is constructed based on a plurality of input feature and output feature pairs, and leaf nodes of the first lookup tree store the input feature and output feature pairs.

[0016] In other words, the input features and output feature pairs of the linear layer in the AI model before optimization are collected and saved in the form of a lookup tree, which facilitates feature lookup in the subsequent AI model inference process.

[0017] Optionally, the lookup tree can be a balanced binary tree to improve the query efficiency of the feature dictionary in the AI model inference process.

[0018] In another possible implementation, the second feature dictionary includes a second lookup tree, the second lookup tree is constructed based on clustering results of a plurality of input feature and output feature pairs, the clustering results include a plurality of clustering clusters and input feature and output feature pairs corresponding to each clustering cluster, and leaf nodes of the second lookup tree store the input feature and output feature pairs corresponding to each clustering cluster.

[0019] In this possible implementation, after a plurality of input feature and output feature pairs of the linear layer are collected, the plurality of input feature and output feature pairs are clustered to obtain a plurality of clustering clusters; and based on the plurality of clustering clusters, a lookup tree is generated.

[0020] Through clustering processing of the plurality of input feature and output feature pairs, the number of input feature and output feature pairs that need to be saved in the feature dictionary is reduced, which on the one hand increases the query speed of the feature dictionary, and on the other hand reduces the memory overhead of the AI model.

[0021] In another possible implementation, the feature dictionary layer is configured to perform a dictionary query on the first feature vector output by the first neural network layer to obtain a second feature vector matching the first feature vector, in a specific implementation, the dictionary query on the first feature vector is performed on the first search tree or the second search tree to obtain a target leaf node of the first search tree or the second search tree, in which the input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity with the output feature vector; and the output feature in the input feature and output feature pair in the target leaf node is output to obtain the second feature vector.

[0022] The feature dictionary query operation replaces the previous matrix multiplication calculation of the linear layer, greatly reduces the calculation overhead of the AI model in the inference process, increases the inference efficiency of the AI model, and reduces the output delay of the AI model.

[0023] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers include a plurality of encoders, at least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer, and the first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

[0024] Compared with the existing transformer architecture-based AI model, the AI model of the present application replaces the entire FFN layer in the original AI model with the feature dictionary layer, thereby eliminating the matrix multiplication calculation of the entire FFN layer in the inference process and replacing it with the feature dictionary query operation which is more computationally efficient, significantly reducing the inference computing power overhead of the AI model, improving the inference efficiency of the AI model, reducing the output delay of the AI model, and improving the user experience.

[0025] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers include a plurality of encoders, at least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer, and the feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer, and a third feature dictionary sub-layer, and the first neural network layer is a scaled dot-product attention mechanism layer; the output of the first feature dictionary sub-layer is connected to the input of the scaled dot-product attention mechanism layer, the output of the scaled dot-product attention mechanism layer is connected to the input of the second feature dictionary sub-layer, and the output of the second feature dictionary sub-layer is connected to the output of the third feature dictionary sub-layer.

[0026] The AI model provided in the application further reduces the inference computing power overhead of the AI model, improves the inference efficiency of the AI model, reduces the output delay of the AI model, and improves the user experience.

[0027] Optionally, the AI model provided in the application is a large model, for example, the AI model can be a large language model, a computer vision (CV) large model, and a multi-modal large model, etc.

[0028] In a second aspect, the application provides an inference device of an AI model, comprising an acquisition module and an inference module, wherein the acquisition module is configured to acquire input data, the input data comprising at least one of text data, voice data, image data and video data; the inference module is configured to input the input data into the AI model to obtain an inference result, the AI model comprising a plurality of feature extraction network layers, at least one of the plurality of feature extraction network layers comprising a first neural network layer and a feature dictionary layer, the first neural network layer being connected to the feature dictionary layer, the feature dictionary layer being configured to perform dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matched with the first feature vector, the second feature vector being used as an output vector of the feature extraction network layer to which the first neural network layer belongs.

[0029] In one possible implementation, the feature dictionary layer comprises a first feature dictionary, the first feature dictionary comprising K dictionary tables, each of the K dictionary tables comprising a plurality of basis vectors, K being a positive integer; and the inference module is specifically configured to: based on the first feature vector, obtain an index value and a weight value of each of the K dictionary tables; based on the index value of each of the K dictionary tables, find a target basis vector corresponding to each of the K dictionary tables from each of the K dictionary tables; and based on the weight value of each of the K dictionary tables, perform weighted summation on the target basis vector corresponding to each of the K dictionary tables to obtain the second feature vector.

[0030] In one possible implementation, the feature dictionary layer further comprises a query sub-layer; and one specific implementation of obtaining the index value and the weight value of each of the K dictionary tables based on the output vector is: taking the output vector as an input of the query sub-layer, and outputting the index value and the weight value of each of the K dictionary tables.

[0031] Optionally, the plurality of basis vectors in each of the dictionary tables and the query sub-layer are trained based on a training sample set.

[0032] In a possible implementation, the feature dictionary layer includes a second feature dictionary, the second feature dictionary is constructed based on a plurality of input feature and output feature pairs, the plurality of input feature and output feature pairs are collected based on inputting the training sample set as an input of an AI model before optimization, and at least one feature extraction network layer of the AI model before optimization includes the first neural network layer and the linear layer.

[0033] In a possible implementation, the second feature dictionary includes a first lookup tree, the first lookup tree is constructed based on the plurality of input feature and output feature pairs, and leaf nodes of the first lookup tree store the input feature and output feature pairs.

[0034] In another possible implementation, the second feature dictionary includes a second lookup tree, the second lookup tree is constructed based on clustering results of the plurality of input feature and output feature pairs, the clustering results include a plurality of clustering clusters and input feature and output feature pairs corresponding to each clustering cluster, and leaf nodes of the second lookup tree store the input feature and output feature pairs corresponding to each clustering cluster.

[0035] In another possible implementation, the inference module is specifically configured to: based on the first feature vector, perform lookup on the first lookup tree or the second lookup tree to obtain a target leaf node of the first lookup tree or the second lookup tree, an input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity with the output vector; and output an output feature in the input feature and output feature pair in the target leaf node to obtain a second feature vector.

[0036] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers include a plurality of encoders, at least one encoder of the plurality of encoders includes the first neural network layer and the feature dictionary layer, and the first neural network layer is a self-attention layer.

[0037] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers include a plurality of encoders, at least one encoder of the plurality of encoders includes the first neural network layer and the feature dictionary layer, and the feature dictionary layer includes a first feature dictionary sublayer, a second feature dictionary sublayer, and a third feature dictionary sublayer, the first neural network layer is a scaled dot-product attention mechanism layer, an output of the first feature dictionary sublayer is connected to an input of the scaled dot-product attention mechanism layer, an output of the scaled dot-product attention mechanism layer is connected to an input of the second feature dictionary sublayer, and an output of the second feature dictionary sublayer is connected to an output of the third feature dictionary sublayer.

[0038] Optionally, the AI model provided in the application is a large model, for example, the AI model can be a large language model, a visual large model, and a multi-modal large model, etc.

[0039] In a third aspect, an embodiment of the present application provides a computing device, comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method of the first aspect is implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0041] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product, which comprises instructions, and when the instructions are executed, the computer executes the method of the first aspect.

[0042] In a sixth aspect, an embodiment of the present application further provides a chip, comprising at least one processor and a communication interface, wherein the processor is configured to execute the method of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0043] FIG. 1 shows a schematic diagram of a transformer architecture;

[0044] FIG. 2 shows a structural schematic diagram of an encoder;

[0045] FIG. 3 shows a structural schematic diagram of an encoder in a large language model provided by an embodiment of the present application;

[0046] FIG. 4 is a flowchart of an inference method of an AI model provided by an embodiment of the present application;

[0047] FIG. 5 shows a construction process schematic diagram of a feature dictionary provided by an embodiment of the present application;

[0048] FIG. 6 shows a structural schematic diagram of an encoder of a large language model provided by an embodiment of the present application;

[0049] FIG. 7 shows a specific structural schematic diagram of a self-attention layer in an encoder;

[0050] FIG. 8 shows a structural schematic diagram of a self-attention layer in an encoder of an AI model provided by an embodiment of the present application;

[0051] FIG. 9 shows a structural schematic diagram after a feedforward neural network layer in an encoder is replaced by a second feature dictionary provided by the present application;

[0052] FIG. 10 shows a structural schematic diagram of an encoder of an AI model provided by an embodiment of the present application;

[0053] FIG. 11 is a structural schematic diagram of an inference device of an AI model provided by an embodiment of the present application;

[0054] FIG. 12 is a structural schematic diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] The term “and / or” mentioned in the present document is a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol “ / ” in the present document represents an or relationship of the associated objects, for example, A / B represents A or B.

[0056] The terms “first” and “second” and the like in the description and claims of the present document are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first feature dictionary and the second feature dictionary are used to distinguish different memory chain data, rather than to describe a specific order of the feature dictionaries.

[0057] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of “exemplary” or “for example” is intended to present concepts in a concrete manner.

[0058] In the description of the embodiments of the present application, unless otherwise specified, “a plurality of” means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0059] With the iterative upgrade of storage hardware (reflected in the increasing storage capacity and the decreasing cost per unit storage), it becomes possible to store more data or model parameters. Therefore, designing a scheme based on feature storage and query to accelerate LLM has great use value and market competitiveness.

[0060] For example, for the retrieval enhancement scheme of FFN in the Transformer layer, most of them construct a learnable parameter table and a hash mapping function (random fixed function or learnable parameters). Although these schemes have good effects in relatively shallow LLMs, they have poor effects in deep LLMs with huge parameters and in scenarios with a large amount of training data.

[0061] In the related art, locality-sensitive hashing (LSH) is used to replace linear layers in a neural network to reduce computation. First, LSH is constructed to replace linear transformation calculation in the FFN, and a strategy is proposed to reduce the update frequency of the hash transformation function in LSH.

[0062] This scheme uses a hash function to replace the linear layer calculation, but a hash bucket representation needs to be constructed based on the hash mapping during the calculation process. Although this scheme proposes a strategy to reduce the update frequency of the hash function, the function is still updated during training, so that the hash representation cannot be stored as a fixed feature.

[0063] Therefore, the present application provides an AI model inference method and device, which uses a feature dictionary that saves a large amount of training data in the training process (such as input and output features of the linear layer) to replace the linear layer in the AI model to optimize the AI model. The optimized AI model in the inference process replaces the linear transformation calculation of the linear layer with the query operation of the feature dictionary, significantly reduces the computational overhead under the premise of ensuring the inference accuracy of the AI model, realizes the inference acceleration of the AI model, reduces the output delay of the AI model, and improves the user experience.

[0064] The AI model inference method and device provided by the embodiments of the present application can solve the problem of the original linear layer in the AI model performing linear calculation that consumes a lot of computing power, such as a large number of matrix multiplication calculations, and instead performing a simple dictionary query operation to determine the original linear layer output according to the query result, thereby ensuring the accuracy of the AI model, significantly reducing the computational overhead, accelerating the inference of the AI model, and reducing the output delay of the AI model.

[0065] The AI model inference method and device provided by the embodiments of the present application have more obvious acceleration effect on AI models with larger linear layer parameter scale, especially when applied to accelerate various large models. For example, the acceleration effect of large language models, visual large models, and multi-modal large models with large-scale parameters is particularly significant.

[0066] The embodiments of the present application take the inference acceleration of large language models as an example to introduce the specific implementation of the AI model inference method and device provided by the embodiments of the present application in detail.

[0067] FIG. 1 shows a schematic diagram of a transformer architecture. As shown in FIG. 1, the transformer architecture includes an encoder block in which a plurality of encoders are stacked, and a decoder block in which a plurality of decoders are stacked. The input of each encoder is the output of the previous encoder, and the input of each decoder includes the output of the previous decoder and the output of the entire encoder block. The input data is processed by the plurality of encoders and the plurality of decoders to obtain the output data. For example, when the transformer architecture is applied to a text translation model, the input data is "machine learning", and after processing by the plurality of encoders and the plurality of decoders in the transformer architecture, the output is "machine learning".

[0068] FIG. 2 shows a structural diagram of an encoder. As shown in FIG. 2, the encoder includes a self-attention layer and a feedforward neural network layer, and the feedforward neural network layer includes a plurality of linear layers for performing linear calculations, such as matrix-by-matrix calculations, on the self-attention layer. Generally, the matrix dimension of the input of the feedforward neural network layer is the same as the matrix dimension of the output.

[0069] The structure of the decoder is similar to that of the encoder, and each decoder also includes a self-attention layer and a feedforward neural network layer. As can be seen from FIG. 1, the AI model based on the transformer architecture has a plurality of encoders and a plurality of decoders, which means that the matrix multiplication calculation of the feedforward neural network layer needs to be performed once for each encoder and each decoder, and the inference calculation process of the entire AI model needs to be performed multiple times. matrix multiplication calculation of the feedforward neural network layer, which requires a lot of calculation overhead and increases the inference time of the AI model. The parameter quantity of the large language model based on the transformer architecture is huge (now reaching hundreds of billions), and the linear calculation quantity involved in the inference process of the large language model will be larger. It is estimated that the ratio of linear calculation quantity in the inference process of the large language model may be more than 50%, therefore, solving the problem of large calculation time overhead of linear calculation of the large language model will have a significant acceleration effect on the inference calculation time of the large language model.

[0070] The embodiment of the present application provides an inference method of an AI model, in which the feedforward neural network layer in each encoder of the large language model is replaced by a feature dictionary layer, and the feature dictionary layer includes a feature dictionary in which the input and output feature information of the feedforward neural network layer is saved. The linear calculation of the feedforward neural network is replaced by the query operation of the feature dictionary, which eliminates the large calculation overhead of linear calculation in the inference calculation process of the large language model, and realizes the inference acceleration of the large language model.

[0071] FIG. 3 shows a structural schematic diagram of an encoder in a large language model provided by an embodiment of the present application. As shown in FIG. 3, the feedforward neural network layer in the encoder of the large language model is replaced by a feature dictionary, and the linear calculation of the original feedforward neural network layer is replaced by the query operation of the feature dictionary, which significantly reduces the computing power overhead of the inference process of the large language model and realizes the inference acceleration of the large language model.

[0072] It should be noted that, considering the hardware performance (such as storage size) of the computing device deploying the large language model and the acceleration demand of the large language model, the feedforward neural network layer in all encoders and decoders in the large language model can be replaced by a feature dictionary, or the feedforward neural network layer in part of the encoders and decoders in the large language model can be replaced by a feature dictionary. The more feedforward neural network layers in the large language model that are replaced by the feature dictionary, the better the inference acceleration effect of the large language model, but the storage overhead of the computing device will also be larger. Conversely, the less feedforward neural network layers in the large language model that are replaced by the feature dictionary, the worse the inference acceleration effect of the large language model, but the storage overhead of the computing device will also be smaller.

[0073] FIG. 4 is a flowchart of an AI model inference method provided by an embodiment of the present application. The method can be executed by any device, apparatus, platform or device cluster with computing capability. The specific computing device executing the method is not specifically limited in the embodiments of the present application, and a suitable computing device can be selected for execution according to the needs. For example, the implementation can be executed on a terminal device, that is, the AI model provided by the embodiments of the present application is deployed on the terminal device, and the inference method of the AI model provided by the embodiments of the present application is executed; the implementation can also be executed on a terminal device and a cloud device (such as a server on the cloud side), that is, the end-cloud collaborative architecture is used to execute the implementation; or the implementation can be completed on a cloud device, and the inference service of the AI model is provided to users in the form of cloud service. In the following, in order to facilitate description, the form of the execution subject is not distinguished, and all are described as the inference device of the AI model. As shown in FIG. 4, the AI model inference method provided by the embodiments of the present application at least includes steps S401 to S404.

[0074] In step S401, input data is obtained.

[0075] Taking an AI model as a large language model and the large language model being deployed on a cloud as an example, when a user has a question and needs to ask, a client (for example, an intelligent question answering client) deployed on a terminal can be opened, the user inputs the question to be asked in an input box of an interactive interface, for example, "please help me write an advertisement for a company (that is, the input data can be text data)", the client sends the input data (that is, the question input by the user) "please help me write an advertisement for a company" to a cloud server on which the large language model is deployed, the cloud server receives the input data to obtain the input data of the user, the cloud server calls the large language model, and takes the input data as an input of the large language model, and the large language model outputs a corresponding answer based on the input data. For example, the output answer is "*** company, let data be safer, and let the future be better".

[0076] In one example, the input data can also be data in other modalities or formats, such as voice data, image data, video data, etc. For example, the input data can be text data, or voice data, or image data, or video data, etc.

[0077] In another example, the input data can also be mixed data in multiple modalities or formats, for example, the input data can be text data + voice data, or text data + image data, or text data + video data, or voice data + image data, or text data + voice data + image data + video data, etc. Embodiments of the present application do not limit the modalities or formats of the input data.

[0078] In step S402, the input data is input into the AI model to obtain an inference result. The AI model includes a plurality of feature extraction network layers, at least one of the plurality of feature extraction network layers includes a first neural network layer and a feature dictionary layer, the first neural network layer is connected to the feature dictionary layer, the feature dictionary layer is used to perform dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matched with the first feature vector, and the second feature vector is used as an output vector of the feature extraction network layer to which the first neural network layer belongs.

[0079] Embodiments of the present application improve the structure of the AI model, replace the linear layer in the feature extraction network layer in the original AI model with the feature dictionary layer, and in the feature extraction process of the feature extraction network, linear calculation which consumes a lot of computing power is not needed or is less needed, and instead, dictionary query operation which saves computing power is performed, so that the computing power consumption is significantly reduced on the basis of ensuring the accuracy of the AI model, the inference of the AI model is accelerated, and the output delay of the AI model is reduced.

[0080] For example, the existing AI model is a large language model based on a transformer architecture, as shown in FIGS. 1 and 2. The large language model includes a plurality of encoders and decoders. Each encoder includes a self-attention layer and a feedforward neural network layer. The feedforward neural network layer includes a plurality of linear layers. The self-attention layer is the first neural network layer. The output of the self-attention layer is input to the feedforward neural network layer. The feedforward neural network layer performs linear computation on the output of the self-attention layer to output a feature vector after encoding by the encoder.

[0081] For example, the existing AI model is a large language model based on a transformer architecture, as shown in FIGS. 1 and 2. The large language model includes a plurality of encoders and decoders. Each encoder includes a self-attention layer and a feedforward neural network layer. The feedforward neural network layer includes a plurality of linear layers. The self-attention layer is the first neural network layer. The output of the self-attention layer is input to the feedforward neural network layer. The feedforward neural network layer performs linear computation on the output of the self-attention layer to output a feature vector after encoding by the encoder.

[0082] In the embodiments of the present application, the feature dictionary that retains the input and output feature information of the feedforward neural network layer can be used to replace the feedforward neural network layer in the existing AI model to obtain a new AI model. When performing inference, the new AI model does not need to perform the linear computation of the feedforward neural network layer, but instead performs a simple feature dictionary query operation, which greatly reduces the inference overhead in the inference process of the AI model and significantly increases the inference speed of the AI model.

[0083] The feature dictionary has a plurality of implementation manners. In a first implementation manner of the feature dictionary, the input features and output features of the feedforward neural network layer are directly stored in pairs to retain the effective information learned by the large language model from a large amount of training corpus.

[0084] Taking a large language model as an example, the training corpus is taken as the input of the large language model. If the large language model of the embodiment of the present application replaces the feedforward neural network layer in each encoder of the large language model, the input feature vector and the output feature vector of the feedforward neural network layer in each encoder are collected to obtain the input feature and the output feature pair of the feedforward neural network layer in each encoder. For example, the input of the feedforward neural network layer in a certain encoder of the large language model is a feature vector A, and the output is a feature vector A', and the input feature and the output feature pair of the feedforward neural network layer are the feature vector A-feature vector A'. Generally speaking, the input feature vector A and the output feature vector A' of the feedforward neural network layer are both multi-dimensional matrices.

[0085] The plurality of input features and output features pairs of each feedforward neural network layer collected constitute the input feature and output feature pair set of each feedforward neural network layer. The feature dictionary corresponding to each feedforward neural network layer is generated according to the input feature and output feature set of each feedforward neural network layer, so as to replace the feedforward neural network layer in the subsequent steps, so that the large language model is optimized, and the inference of the large language model is accelerated.

[0086] In the inference process of the large language model, the output vector of the first neural network layer (i.e., the self-attention layer) connected to the feedforward neural network layer is used to query the feature dictionary to obtain the output feature vector matched with the output vector. Here, the meaning of matching is that the output vector is taken as the query input feature vector, then the similarity between the input feature vector and the input feature vector A in the input feature and output feature pair in the feature dictionary is calculated, and the input feature vector A with the highest similarity is found out. The output feature vector A` corresponding to the input feature vector A in the input feature and output feature pair is the output feature vector matched with the output vector.

[0087] The second implementation manner of the feature dictionary is to represent the effective information learned by the large language model from a large amount of training corpus through learnable parameters. For example, the feature dictionary in this implementation manner can include a plurality of dictionary tables, each dictionary table includes a plurality of basis vectors, and the basis vectors in each dictionary table in the feature dictionary are trained and adjusted through the plurality of input features and output features, so that the output feature vector can be represented by the combination of the basis vectors in each dictionary table.

[0088] For example, the feature dictionary that needs to be learned is replaced in the feedforward neural network layer in the encoder of the large language model, and then the large language model is trained. In the training process, the basis vectors of each dictionary table in the feature dictionary are adjusted, so that the feature dictionary learns to represent the output of the original feedforward neural network layer by the combination of the basis vectors.

[0089] The generation methods of the two feature dictionaries are described below.

[0090] The first generation method of the feature dictionary is to store a plurality of input feature and output feature pairs in a form conducive to query. For example, the plurality of input feature and output feature pairs are stored in the form of a search tree. Specifically, the plurality of input feature and output feature pairs can be stored in the form of a balanced binary tree. When querying the feature dictionary, the balanced binary tree is used to quickly find the corresponding query result, i.e., the output feature vector.

[0091] In another example, to further increase the query efficiency of the feature dictionary and reduce the storage overhead of the feature dictionary, after obtaining a plurality of input feature and output feature pairs of a feedforward neural network layer, the plurality of input feature and output feature pairs are clustered and compressed. Then, the balanced binary tree is constructed using the clustered and compressed data.

[0092] FIG. 5 shows a construction process of the feature dictionary according to an embodiment of the present application. As shown in FIG. 5, for the construction of the feature dictionary of any layer of the feedforward neural network, first, the training corpus is input into the large language model trained, and then the input feature vector and the output feature vector of the feedforward neural network layer are collected, to obtain a plurality of input feature and output feature pairs of the feedforward neural network layer. Then, the plurality of input feature and output feature pairs are clustered using a preset clustering algorithm, to obtain a plurality of clustering clusters. The input feature and output feature pair of the clustering cluster center represents the input feature and output feature pair in the clustering cluster. Finally, the balanced binary tree is constructed using the input feature and output feature pairs corresponding to the plurality of clustering clusters, i.e., the input feature and output feature pairs corresponding to the plurality of clustering clusters are stored in each leaf node of the balanced binary tree according to the construction rule of the balanced binary tree.

[0093] In the embodiment of the present application, the input feature vectors in the plurality of input feature and output feature pairs are clustered, and the clustering result of the input feature vectors represents the clustering result of the plurality of input feature and output feature pairs. For example, the input feature and output feature pair is an input feature vector A-output feature vector A', when the clustering algorithm is executed, the input feature vector A is clustered and analyzed, to obtain a plurality of clustering clusters. The input feature vector A at the center of the clustering cluster represents the clustering cluster, to obtain the input feature vectors A corresponding to the plurality of clustering clusters, and further obtain the input feature vector A-output feature vector A' pairs corresponding to the plurality of clustering clusters. Finally, the input feature vector A-output feature vector A' pairs are stored in each leaf node of the balanced binary tree according to the construction rule of the balanced binary tree.

[0094] The clustering processing can significantly reduce the number of the plurality of input feature and output feature pairs, and further reduce the storage overhead and increase the query efficiency.

[0095] It should be noted that the embodiments of this application do not specifically limit which clustering algorithm is used to cluster multiple input and output feature pairs. For example, the clustering algorithm used in the embodiments of this application can be K-means clustering algorithm, hierarchical clustering algorithm, and DBSCAN clustering algorithm, etc., and a suitable clustering algorithm can be selected according to actual needs.

[0096] The second implementation method of the feature dictionary is as follows: the feature dictionary stores learnable parameters, the feature dictionary layer is replaced with the linear layer in the existing AI model, a new AI model is constructed, and then the AI ​​model is trained. During the training process of the AI ​​model, the parameters of the feature dictionary are adjusted.

[0097] For example, we can learn the parameters of the feature dictionary using dictionary learning. We learn multiple basis vectors from each table in the feature dictionary, and then use a weighted sum of these basis vectors to represent the feature vector that matches the output of the first neural network.

[0098] The feature dictionary includes multiple dictionary tables B K Each dictionary table includes multiple learnable basis vectors. The goal of dictionary learning is to learn from multiple dictionary tables B in the feature dictionary. K The basis vectors in the dictionary represent the output feature vectors. The specific dictionary learning process is as follows: the first feature vector X output by the first neural network is used as the input to the feature dictionary. The feature dictionary first uses the input X as the input to the function f(X) (i.e., the query sublayer) to obtain the index values ​​I of each dictionary table. K and the weight C of each dictionary table K Then, using the index values ​​I of each dictionary table K The query retrieves the target basis vectors from each dictionary table. Then utilize the weight value C of each dictionary table K The target basis vectors output by each dictionary table The weighted summation is performed to obtain the output y of the feature dictionary, which is the second feature vector. The second feature vector is then used as the input of the subsequent neural network layer until the output of the AI ​​model is obtained. The output of the AI ​​model is used to adjust the basis vectors and parameters of the function f(X) in each dictionary table of the feature dictionary to obtain the trained feature dictionary.

[0099] This application does not impose specific limitations on the specific dictionary learning algorithm used. Appropriate dictionary learning algorithms can be selected as needed to learn the feature dictionary of this application embodiment.

[0100] The feature dictionary is generated by dictionary learning method, and the output feature vector is represented by a few base vectors, which realizes dimension reduction calculation in a certain sense and speeds up the calculation. For example, for a set of three-dimensional vectors, we can represent them with a three-dimensional orthogonal coordinate system (x, y, z three unit vectors). The three vectors can represent all three-dimensional vectors by combination, which is the learning goal of the base vectors in the feature dictionary. All output feature vectors in the output feature and output feature pair can be represented by the base vectors in the dictionary table.

[0101] It should be noted that the large language model provided by the embodiments of the present application can be obtained by modifying the structure of an existing large language model, for example, replacing the feedforward neural network layer in the encoder of the original large language model with the second feature dictionary to obtain a large language model with a structure that has a faster inference speed. When the original large language model is a trained large language model, the parameter-learnable feature dictionary is replaced with the feedforward neural network in the large language model, and then the large language model is trained using training corpus. At this time, the other parameters of the large language model are frozen and only the parameters of the feature dictionary are adjusted (including the base vectors of each dictionary table in the feature dictionary and the query sublayer), to obtain a trained feature dictionary. In another example, the original large language model can also be an untrained large language model. At this time, the parameter-learnable feature dictionary is replaced with the feedforward neural network in the large language model, and then the large language model is trained using training corpus. The feature dictionary and the large language model are learned together, that is, the parameters of the other layers in the large language model and the parameters of the feature dictionary need to be learned and adjusted during the training process. After the training of the large language model is completed, the trained feature dictionary is also obtained.

[0102] The feature dictionary obtained in the foregoing steps is used to replace the feedforward neural network in the encoder and / or decoder of the large language model. In this way, during the inference calculation process of the large language model, linear calculation (such as matrix multiplication calculation) with high computational power is not required, and instead, feature dictionary query operation with smaller calculation overhead is performed, which reduces the computational power overhead in the inference calculation process of the entire large language model, speeds up the inference speed, and further reduces the response output delay of the large language model, and improves the user experience.

[0103] FIG. 6 shows a structural schematic diagram of an encoder of a large language model provided by the embodiments of the present application.

[0104] When the feature dictionary is the first feature dictionary, that is, the feature dictionary that uses a balanced binary tree to store the input feature and the output feature pair, the processing process of the encoder in the large language model is: taking the output of the self-attention layer as the input of the feature dictionary, using the input vector (that is, the output vector of the self-attention layer) to search the balanced binary tree, according to the similarity between the input vector and the stored feature (that is, the input feature in the input feature and the output feature pair stored on the balanced binary tree), searching from the root node of the balanced binary tree to the leaf node to determine the target leaf node, and outputting the output feature vector stored on the target leaf node to obtain the output feature vector, and the output feature vector is taken as the input of the next encoder for further processing.

[0105] It should be noted that there are various similarity algorithms for calculating the similarity between the input vector and the stored feature, such as Euclidean distance, Manhattan distance, Chebyshev distance, cosine similarity, and Jaccard distance, and the appropriate similarity algorithm can be selected according to the actual situation to calculate the similarity between the input vector and the stored feature, and the similarity algorithm used in the embodiments of the present application is not limited.

[0106] FIG. 7 shows a specific structure diagram of the self-attention layer in the encoder. As shown in FIG. 7, the encoder includes a linear layer, a scaled dot-product attention layer, a feature fusion layer (concat), and a linear layer, and it can be found that the self-attention layer also includes multiple linear layers, which also consume a lot of computing power. Therefore, in order to further increase the inference speed of the large language model, the linear layers in the self-attention layer are also replaced by the feature dictionary.

[0107] According to the feature dictionary construction algorithm described above, the feature dictionary corresponding to the linear layer in the self-attention layer is constructed, and then the linear layer in the self-attention layer is replaced by the constructed feature dictionary, further reducing the computing power consumption of the large language model and increasing the inference speed of the large language model.

[0108] FIG. 8 shows a structure diagram of the self-attention layer in the encoder of the AI model provided by the embodiments of the present application. As shown in FIG. 8, based on the feature dictionary constructed by the embodiments of the present application, the linear layers in the original self-attention layer are replaced, and in the inference process, the query operation of the feature dictionary replaces the matrix multiplication calculation of the previous linear layer, reducing the computing power consumption and realizing the inference acceleration of the entire large language model.

[0109] Figure 9 shows a schematic diagram of the structure after replacing the feedforward neural network layer in the encoder with the second feature dictionary provided in this application. As shown in Figure 9, by replacing the feedforward neural network in the encoder of the large language model with the feature dictionary constructed in the embodiment of this application, the original calculation process of the feedforward neural network is replaced by a feature dictionary lookup operation, effectively reducing the amount of computation and improving the inference speed of the large language model. The model structure after feature dictionary replacement can be called Dict-LLM. The inference calculation process of the AI ​​model provided in the embodiment of this application is as follows: the output of each encoder after passing through the self-attention layer is used as the input of the feature dictionary. As mentioned above, the feature dictionary includes multiple dictionary tables B. K The specific feature dictionary lookup process is as follows: First, the input X is used as the input to the function f(X), and the index values ​​I of each dictionary table are obtained. K and the weight C of each dictionary table K Then, using the index value I of each dictionary table K The query retrieves the target basis vectors from each dictionary table. Then utilize the weight value C of each dictionary table K The target basis vectors output by each dictionary table The weighted summation yields the output y of the feature dictionary. The specific calculation formula is shown below:

[0110] Figure 10 shows a schematic diagram of the encoder structure of the AI ​​model provided in this application embodiment. As shown in Figure 10, the feature dictionary constructed in this application embodiment replaces all linear layers (including linear layers in the self-attention layer and feedforward neural network layers connected to the self-attention layer) in the encoder of the large language model with the feature dictionary. The inference process of the large language model provided in this application embodiment is as follows: In each encoder, the input X of the encoder is received (the input of the first encoder layer is the output of the embedded representation layer, and the input of the encoder of the subsequent layers is the output of the previous encoder layer), the input X is input to the feature dictionary for querying, and the output vectors Q, K, and V are output, where the expression of the Q vector is Q = DictLLM. Q (X), the expression for the K vector is K = DictLLM K The expression for vector (X), V, is V = DictLLM. V (X). After obtaining the Q vector, K vector, and V vector through the feature dictionary lookup, they are input into the softmax layer for calculation. The calculation result is then used as the input to the feature dictionary, and the output is the original output of the attention layer. This output is then used as the input to the feature dictionary, and the original output feature vector of the feedforward neural network layer is output. This output feature vector is the output of the encoder (i.e., the feature vector output by the encoder).

[0111] Based on the feature vectors output by several feature extraction network layers, determine the inference result of the AI model. Taking an AI model based on the transformer architecture as an example, after being processed by multiple encoder and decoder layers, the feature vectors of the input data are extracted, and then the feature vectors are input into a classifier to output the inference result of the AI model. For example, if the AI model based on the transformer architecture is a translation model and the input data is the Chinese text sequence "machine learning", then multiple encoders and decoders in the AI model process the input data and output the feature vector Y representing the features of the input data "machine learning". Taking this feature vector Y as the input of the classifier, the classifier outputs the probability distribution of each word based on the feature vector Y, and the inference result output based on this probability distribution is the English text sequence "machine learning" corresponding to the Chinese text sequence.

[0112] It should be noted that the inference result is related to the application scenario of the AI model. For example, in the above example, the AI model is a translation model applied to the translation scenario, and the input data is Chinese text data, then the inference result of the AI model is the English translation text corresponding to the Chinese text data. Another example is that if the AI model is a large language model applied to the question and answer scenario and the input data is the question text, then the inference result of the AI model is the answer text corresponding to the question text.

[0113] The inference method provided in the embodiments of this application replaces the relatively computationally expensive linear layers in the large language model with feature dictionaries, significantly reducing the computational overhead in the inference process of the large language model, increasing the inference speed of the large language model, reducing the response delay of the large language model, and improving the user experience.

[0114] It should be noted that the first type of feature dictionary stores a large number of features (directly stores the input feature and output feature pairs of the linear layer), so it requires a large storage overhead, but the acceleration effect is good. Through experiments, it is confirmed that using the first type of feature dictionary provided in the embodiments of this application to optimize and accelerate the inference of the large language model, the computational overhead of the query operation of the feature dictionary in the inference process of the large language model is only 1 / 1000 of the computational overhead of the original feed-forward neural network, but the output accuracy of the model remains unchanged. Therefore, when using the first type of feature dictionary for inference optimization and acceleration, it is recommended that the storage space of the hardware for deploying the large language model be more than 100TB. Otherwise, it is recommended to use the second type of feature dictionary to optimize and accelerate the large language model. The second type of feature dictionary has a smaller storage overhead for the device. It is experimentally confirmed that the large language model optimized with the second type of feature dictionary has only eight times the number of parameters compared to the original model, but the computational amount is only 1 / 30 of the original model, and the output accuracy of the model remains unchanged.

[0115] For example, the calculation cost of the feature dictionary is n*log(N)*d after the model is optimized using the first feature dictionary (i.e., the feature dictionary constructed by the balanced binary tree), while the calculation cost of the original model is n*4d 2 , where N is the size of the data set (1.4T), n is the length of the sentence (2k), and d is the dimension of the model features (10k). Through calculation, it can be obtained that the calculation amount of the query scheme of the first feature dictionary is only 1 / 1000 of the calculation amount of the original feedforward neural network, that is, 1000 times acceleration is achieved. We conducted experiments on a 1B model, replaced 3 layers of feedforward neural network with a feature dictionary, and the model output accuracy remained unchanged.

[0116] For example, for user input data: What should I do if I have a headache?

[0117] The output of the existing large language model is: If your headache is due to your head pain, it is recommended that you first go to a doctor. If your headache is caused by excessive work pressure, lack of sleep, irregular diet, etc., you can try the following methods to relieve the headache:

[0118] 1. Maintain adequate sleep and try to sleep 8-9 hours a day. 2. Maintain a healthy diet and try to avoid excessive intake of sugar and caffeine. 3. Exercise moderately, which can help relieve stress and anxiety.

[0119] The output of the large language model optimized by the first feature dictionary is: First, consult a doctor or clinic to understand the cause of the pain and the solution. Second, you can try the following measures: 1. Rest: Rest to relieve pain. 2. Drink water: Drinking more water can help relieve pain. 3. Diet: Avoid greasy, spicy, and irritating foods and maintain a balanced diet. 4. Drug treatment: If the pain persists, consider using painkillers. 5. Physical therapy: You can try physical therapy such as massage, hot compress, etc.

[0120] For the large language model optimized by the second feature dictionary, we conducted experiments on five natural language processing tasks: PIQA, WinoGande, WSC, ARC-Easy, and ARC-Challenge. Our scheme reduced the calculation amount of the feedforward neural network layer by 30 times while ensuring almost unchanged model accuracy.

[0121] The experimental data is shown in the following table:

[0122] Of course, the inference method of the AI model provided in the embodiments of the present application can also be applied to scenarios other than large language models, for example, it can be applied to inference acceleration of convolutional neural networks (CNN).

[0123] For example, a convolutional neural network generally includes operations of four kinds of layers, including a convolution layer, a rectified linear unit (Relu) layer, a pooling layer, and a fully connected (FC) layer.

[0124] In a convolutional neural network model, multiple convolution layers, multiple activation function layers, and multiple pooling layers are alternately included, and finally data is input to a fully connected layer to obtain an output result. For example, input image-Conv-Relu-Conv-Relu-Pooling-Conv-Relu-Conv-Relu-Pooling-Conv-Relu-Conv-Relu-Pooling-FC-output.

[0125] The role of the convolution layer is to identify features of input image data through multiple filters, each filter having a scanning range for scanning data information of a certain region of the input image. The calculation result of the current convolution layer is input to the next layer (which can be an activation function layer, a pooling layer, or a fully connected layer) for processing.

[0126] The role of the activation function layer is to perform an operation similar to MAX(0, x) on input image data, that is, to compare each value in the input image data with the 0 value, and if it is greater than the 0 value, it is retained, and if it is less than the 0 value, it is set to 0. The activation function layer provides the sparsity of the input image data (i.e., the percentage of 0 values in the total number of data), and does not change the size (i.e., the amount of data) of the input image data.

[0127] The role of the pooling layer is down-sampling, that is, extracting data by skipping rows or columns in the two-dimensional matrix of each layer of input data, thereby reducing the size of the input image data.

[0128] The operation process of the fully connected layer is similar to that of the convolution layer, except that the filter of the fully connected layer does not scan a small region of the input image data, but scans the entire input image data at once, and then outputs a value. There are multiple filters in the fully connected layer, corresponding to multiple different very specific image features. The output value is equivalent to a "score" indicating the "likelihood" of the input image data containing these features.

[0129] Generally, the calculation amount of the convolution layer and the fully connected layer in a convolutional neural network model accounts for more than 90% of the total calculation amount of the convolutional neural network model, and the convolution layer is a variant of the linear layer and belongs to the category of the linear layer, and the fully connected layer is composed of multiple linear layers. Therefore, the existing convolutional neural network model can be optimized by using the method provided in the embodiments of the present application, and the feature dictionary that retains a large amount of effective information (such as the input and output features of the convolution layer, or the learnable parameters such as the basis vectors in multiple dictionary tables) in the training process is used to replace the convolution layer in the convolutional neural network model to optimize the convolutional neural network model. After such optimization, the convolutional neural network model, in the inference process, the query operation of the feature dictionary replaces the convolution operation of the convolution layer, significantly reduces the computing power consumption, realizes the inference acceleration of the convolutional neural network model, reduces the output delay of the convolutional neural network model, and improves the user experience.

[0130] In another example, to further accelerate the inference of the convolutional neural network model, all convolution layers and fully connected layers in the convolutional neural network model are replaced by the feature dictionary layer of the embodiments of the present application. In this way, in the inference process of the convolutional neural network model, the query operation of the feature dictionary, which is relatively energy-saving, replaces the convolution operation of the convolution layer and the linear calculation operation of the fully connected layer, which are relatively energy-consuming, thereby significantly reducing the computing power consumption and realizing the inference acceleration of the convolutional neural network model.

[0131] The detailed implementation of the feature dictionary can be referred to the description above, and will not be repeated here for brevity.

[0132] Based on the same idea as the foregoing embodiment of the inference method of the AI model, the embodiments of the present application also provide an AI model inference device 1100, which can be deployed in a terminal device or a cloud server to improve the inference speed of the AI model. The AI model inference device 1100 includes units or modules to implement each step of the AI model inference method shown in FIGS. 4-10.

[0133] FIG. 11 is a structural schematic diagram of an inference device of an AI model according to an embodiment of the present application. As shown in FIG. 11, the inference device 1100 of the AI model at least includes an obtaining module 1101, an input module 1102, and an inference module 1103. The obtaining module 1101 is configured to obtain input data, the input data including at least one of text data, voice data, image data, and video data. The inference module 1102 is configured to input the input data into an AI model to obtain an inference result. The AI model includes a plurality of feature extraction network layers. At least one of the plurality of feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is configured to perform a dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matched with the first feature vector. The second feature vector is used as an output vector of the feature extraction network layer to which the first neural network layer belongs.

[0134] In one possible implementation, the feature dictionary layer includes a first feature dictionary. The first feature dictionary includes K dictionary tables. Each of the K dictionary tables includes a plurality of basis vectors. K is a positive integer. The inference module 1103 is specifically configured to: based on the first feature vector, obtain an index value and a weight value of each of the K dictionary tables; based on the index value of each of the K dictionary tables, find a target basis vector corresponding to each of the K dictionary tables from each of the K dictionary tables; and based on the weight value of each of the K dictionary tables, perform a weighted summation on the target basis vector corresponding to each of the K dictionary tables to obtain the second feature vector.

[0135] In one possible implementation, the feature dictionary layer further includes a query sub-layer. One specific implementation of obtaining the index value and the weight value of each of the K dictionary tables based on the output vector is that: the output vector is taken as an input of the query sub-layer, and the index value and the weight value of each of the K dictionary tables are output.

[0136] Optionally, the plurality of basis vectors in each of the dictionary tables and the query sub-layer are trained based on a training sample set.

[0137] In one possible implementation, the feature dictionary layer includes a second feature dictionary. The second feature dictionary is constructed based on a plurality of input feature and output feature pairs. The plurality of input feature and output feature pairs are obtained by collecting input features and output features of a linear layer after the training sample set is input into an AI model before optimization. At least one of the feature extraction network layers of the AI model before optimization includes the first neural network layer and the linear layer.

[0138] In one possible implementation, the second feature dictionary includes a first search tree. The first search tree is constructed based on the plurality of input feature and output feature pairs. Leaf nodes of the first search tree store the input feature and output feature pairs.

[0139] In another possible implementation, the second feature dictionary comprises a second search tree, the second search tree is constructed based on clustering results of the plurality of input features and the plurality of pairs of output features, the clustering results comprise a plurality of clustering clusters and pairs of input features and output features corresponding to each clustering cluster, and leaf nodes of the second search tree store the pairs of input features and output features corresponding to each clustering cluster.

[0140] In another possible implementation, the inference module 1103 is specifically configured to: based on the first feature vector, search the first search tree or the second search tree to obtain a target leaf node of the first search tree or the second search tree, an input feature in a pair of input features and output features stored in the target leaf node has the highest similarity with the output vector; and output an output feature in the pair of input features and output features in the target leaf node to obtain a second feature vector.

[0141] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers comprise a plurality of encoders, at least one encoder of the plurality of encoders comprises a first neural network layer and a feature dictionary layer, and the first neural network layer is a self-attention layer.

[0142] In another possible implementation, the AI model is a transformer architecture-based AI model, and the plurality of feature extraction network layers comprise a plurality of encoders, at least one encoder of the plurality of encoders comprises a first neural network layer and a feature dictionary layer, and the feature dictionary layer comprises a first feature dictionary sublayer, a second feature dictionary sublayer, and a third feature dictionary sublayer, and the first neural network layer is a scaled dot-product attention mechanism layer; an output of the first feature dictionary sublayer is connected to an input of the scaled dot-product attention mechanism layer, an output of the scaled dot-product attention mechanism layer is connected to an input of the second feature dictionary sublayer, and an output of the second feature dictionary sublayer is connected to an output of the third feature dictionary sublayer.

[0143] Optionally, the AI model provided in the embodiments of the present application is a large model, for example, the AI model can be a large language model, a visual large model, and a multi-modal large model, and the like.

[0144] The inference apparatus 1100 of the AI model according to the embodiments of the present application can correspond to performing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module in the inference apparatus 1100 for the AI model are respectively for realizing the corresponding processes of each method in FIGS. 4-10, and for brevity, will not be described here.

[0145] The embodiments of the present application also provide a computing device comprising at least one processor, a memory, and a communication interface, the processor being configured to execute the methods described in FIGS. 4-10.

[0146] FIG. 12 is a structural schematic diagram of a computing device provided in an embodiment of the present application.

[0147] As shown in FIG. 12, the computing device 1200 includes at least one processor 1201, a memory 1202, and a communication interface 1203. The processor 1201, the memory 1202, and the communication interface 1203 are communicatively connected, which can be achieved by a wired (for example, a bus) manner or a wireless manner. The communication interface 1203 is configured to send and / or receive data sent by other devices; the memory 1202 stores computer instructions, and the processor 1201 executes the computer instructions to execute the method in the foregoing method embodiments, so as to accelerate the inference of an AI model, significantly reduce the calculation overhead of the AI model in the inference process on the basis of ensuring the model precision, realize the inference acceleration of the AI model, and reduce the output delay of the AI model.

[0148] It should be understood that, in the embodiments of the present application, the processor 1201 can be a central processing unit CPU, and the processor 1201 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0149] The memory 1202 can include read-only memory and random access memory, and provide instructions and data for the processor 1201. The memory 1202 can also include non-volatile random access memory.

[0150] The memory 1202 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Where the nonvolatile memory is a read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available, for example, static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). The storage device 1204 can be a nonvolatile memory or storage device such as a magnetic disk, optical disk, flash memory, or tape. The storage device 1204 can also include a non-transitory computer-readable medium in which computer instructions can be stored.

[0151] It should be understood that the computing device 1200 according to the embodiments of the present application can perform the method shown in FIGS. 4-10 of the embodiments of the present application, and detailed description of the method implemented by the method is described above, and for brevity, will not be repeated here.

[0152] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the above-mentioned method is implemented.

[0153] The embodiments of the present application provide a chip, which includes at least one processor and an interface, the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the above-mentioned method.

[0154] The embodiments of the present application provide a computer program or computer program product, which includes instructions, when the instructions are executed, the computer executes the above-mentioned method.

[0155] Those skilled in the art should further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0156] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0157] The above detailed description of the specific implementation is further detailed for the purpose of the present application, technical solutions and beneficial effects. It should be understood that the above description is only a specific implementation of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A reasoning method for an AI model, characterized in that, include: Acquire input data, which includes at least one of text data, voice data, image data, and video data; The input data is input into an AI model to obtain a reasoning result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

2. The method according to claim 1, characterized in that, The feature dictionary layer includes a first feature dictionary, which includes K dictionary tables. Each of the K dictionary tables includes multiple basis vectors, where K is a positive integer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector, including: Based on the first feature vector, the index value and weight value of each dictionary table in the K dictionary tables are obtained; The target basis vector corresponding to each dictionary table is obtained by searching the dictionary table based on the index value of each dictionary table; The second feature vector is obtained by weighting and summing the target basis vectors corresponding to each dictionary table based on the weight values ​​of each dictionary table.

3. The method according to claim 2, characterized in that, The feature dictionary layer also includes a query sublayer; The process of obtaining the index value and weight value of each of the K dictionary tables based on the output vector includes: The output vector is used as the input to the query sub-layer, and the index value and weight value of each of the K dictionary tables are output.

4. The method according to claim 3, characterized in that, The multiple basis vectors in each dictionary table and the query sub-layer are obtained based on the training sample set.

5. The method according to any one of claims 1-4, characterized in that, The feature dictionary layer includes a second feature dictionary, which is constructed based on multiple input and output feature pairs. The multiple input and output feature pairs are obtained by collecting the input and output features of the linear layer after using the training sample set as the input of the AI ​​model before optimization. The at least one feature extraction network layer of the AI ​​model before optimization includes the first neural network layer and the linear layer.

6. The method according to claim 5, characterized in that, The second feature dictionary includes a first search tree, which is constructed based on the plurality of input feature and output feature pairs, and the leaf nodes of the first search tree store the input feature and output feature pairs.

7. The method according to claim 5, characterized in that, The second feature dictionary includes a second search tree, which is constructed based on the clustering results of the multiple input feature and output feature pairs. The clustering results include multiple clusters and the input feature and output feature pairs corresponding to each cluster. The leaf nodes of the second search tree store the input feature and output feature pairs corresponding to each cluster.

8. The method according to claim 6 or 7, characterized in that, The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector, including: Based on the first feature vector, a search is performed on the first search tree or the second search tree to obtain the target leaf node of the first search tree or the second search tree. The input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity to the output vector. The output feature of the input feature and output feature pair in the target leaf node is output to obtain the second feature vector.

9. The method according to any one of claims 1-8, characterized in that, The AI ​​model is an AI model based on the transformer architecture. The plurality of feature extraction network layers include a plurality of encoders. At least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer. The first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

10. The method according to any one of claims 1-8, characterized in that, The AI ​​model is an AI model based on the transformer architecture. The plurality of feature extraction network layers include a plurality of encoders. At least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer. The feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer and a third feature dictionary sub-layer. The first neural network layer is a scaled dot product attention mechanism layer. The output of the first feature dictionary layer is connected to the input of the scaled dot product attention mechanism layer, the output of the scaled dot product attention mechanism layer is connected to the input of the second feature dictionary layer, and the output of the second feature dictionary layer is connected to the output of the third feature dictionary layer.

11. The method according to any one of claims 1-10, characterized in that, The AI ​​model includes any one of the following: large language model, visual large language model, and multimodal large model.

12. An inference device for an AI model, characterized in that, include: An acquisition module is used to acquire input data, which includes at least one of text data, voice data, image data, and video data; The inference module is used to input the input data into the AI ​​model to obtain the inference result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

13. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-11 to be implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-11 to be implemented.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the method as described in any one of claims 1-11 to be implemented.

Images