Ai model inference method and apparatus

Abstract

Provided are an AI model inference method and apparatus. The method comprises: acquiring input data; and inputting the input data into an AI model to obtain an inference result, wherein the AI model comprises a plurality of feature extraction network layers, at least one feature extraction network layer among the plurality of feature extraction network layers comprises a first neural network layer and a feature dictionary layer, the first neural network layer is connected to the feature dictionary layer, and the feature dictionary layer is used for performing dictionary query on a first feature vector output by the first neural network layer to obtain a second feature vector matching the first feature vector. In the AI model provided in the present application, a feature dictionary query operation replaces previous linear calculation that is computationally intensive, thereby significantly reducing computational overhead of the AI model during inference while ensuring model accuracy, implementing inference acceleration of the AI model, and reducing an output delay of the AI model.

Description

A reasoning method and apparatus for an AI model

[0001] This application claims priority to Chinese patent application No. 202410836122.8, filed on June 25, 2024, entitled "A Reasoning Method and Apparatus for an AI Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a reasoning method and apparatus for an AI model. Background Technology

[0003] Large language models (LLMs) based on Transformers have been widely used and have achieved great success in general and specialized domain tasks. However, with the development of LLMs, the number of model parameters has increased rapidly (for example, the number of LLM model parameters has reached hundreds of billions). The computational cost of the LLM model inference process is increasing, and the inference time is getting longer and longer, resulting in increased response latency and a poor user experience. Summary of the Invention

[0004] The embodiments of this application provide an AI model inference method and apparatus. By replacing the previously computationally intensive linear calculation with a feature dictionary lookup operation, the computational overhead of the AI ​​model during the inference process is significantly reduced while ensuring the model's accuracy, thereby accelerating the inference process of the AI ​​model and reducing the output latency of the AI ​​model.

[0005] In a first aspect, this application provides a reasoning method for an AI model, including acquiring input data, which includes at least one of text data, voice data, image data, and video data; inputting the input data into an AI model to obtain a reasoning result, wherein the AI ​​model includes several feature extraction network layers, at least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer, the first neural network layer is connected to the feature dictionary layer, the feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector, and the second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

[0006] The AI ​​model inference method provided in this application replaces the linear layer in the original AI model with a feature dictionary layer. In this way, the matrix multiplication calculation of the original linear layer is replaced by the query operation of the feature dictionary during the AI ​​model inference process, which significantly reduces the computational overhead of the AI ​​model inference process, accelerates the AI ​​model inference, reduces the output latency of the AI ​​model, and the accuracy of the AI ​​model is almost not reduced.

[0007] Taking a large language model based on the transformer architecture as an example, the feature extraction network in the large language model includes multiple encoders, which can be referred to as multiple feature extraction network layers. At least one encoder (to distinguish it from other encoders, it is called a feature dictionary encoder) includes a self-attention layer (i.e., the first neural network layer) and a feature dictionary layer connected to the self-attention layer. Compared with existing encoders that include a self-attention layer and a feedforward neural network layer, the feature dictionary encoder replaces the feedforward neural network layer. It does not need to perform computationally intensive linear calculations. Instead, it performs a more computationally efficient dictionary lookup operation to obtain the output feature vector of the encoder.

[0008] In a large language model, multiple encoders extract features from the input data (which typically passes through an embedding layer before the first encoder to obtain an embedding vector). The feature vector output by the last encoder is the final feature vector. For example, if the large language model includes a first encoder, a second encoder, and so on up to the Nth encoder, the first encoder extracts features from the embedding vector output by the embedding layer. The second encoder updates the feature vector output by the first encoder, outputting the updated feature vector. The Nth encoder updates the feature vector output by the (N-1)th encoder, outputting the final feature vector. When the feature dictionary encoder is the last encoder (i.e., the Nth encoder), it includes a self-attention layer and a feature dictionary layer connected to it. The output of the self-attention layer is the first feature vector, and the output of the feature dictionary layer is the second feature vector. This second feature vector is the output of the feature dictionary encoder, which is the final feature vector.

[0009] In one possible implementation, the feature dictionary layer includes a first feature dictionary, which comprises K dictionary tables. Each of the K dictionary tables includes multiple basis vectors, where K is a positive integer. A specific implementation of the feature dictionary layer performing dictionary lookups on the first feature vector output by the first neural network layer to obtain a second feature vector matching the first feature vector is as follows: based on the first feature vector, the index and weight values ​​of each of the K dictionary tables are obtained; based on the index values ​​of each dictionary table, the target basis vector corresponding to each dictionary table is retrieved; based on the weight values ​​of each dictionary table, the target basis vectors corresponding to each dictionary table are weighted and summed to obtain the second feature vector.

[0010] The output of the original linear layer (such as a feedforward neural network layer) is represented by a combination of multiple basis vectors (i.e., a weighted sum of multiple basis vectors), which reduces the computational overhead in the AI ​​model inference process and speeds up the inference speed of the AI ​​model.

[0011] In one possible implementation, the feature dictionary layer also includes a query sublayer; a specific implementation of obtaining the index values ​​and weight values ​​of each of the K dictionary tables based on the output vector is as follows: the output vector is used as the input of the query sublayer, and the index values ​​and weight values ​​of each of the K dictionary tables are output.

[0012] Optionally, multiple basis vectors and query sublayers in each dictionary table are trained based on the training sample set. In other words, the input and output information of the original linear layers in the AI ​​model can be stored as learnable parameters, which can be learned through dictionary learning. The output of the original linear layers (such as linear layers in feedforward neural networks or self-attention layers) is represented by combinations of basis vectors. During AI model inference, there is no need to perform computationally intensive linear calculations; instead, computationally efficient dictionary lookup operations are performed, effectively reducing computational overhead and increasing the inference speed of the AI ​​model.

[0013] In one possible implementation, the feature dictionary layer includes a second feature dictionary, which is constructed based on multiple input and output feature pairs. These multiple input and output feature pairs are obtained by collecting the input and output features of the linear layer after using the training sample set as input to the AI ​​model before optimization. The AI ​​model before optimization has at least one feature extraction network layer, which includes a first neural network layer and a linear layer.

[0014] It should be explained that the AI ​​model before optimization here is relative to the AI ​​model of this application. The AI ​​model before optimization and the AI ​​model of this application belong to the same type of AI model. The difference is that the AI ​​model before optimization uses the existing structure of AI models. For example, both the AI ​​model before optimization and the AI ​​model of this application are large language models based on the transformer architecture. The AI ​​model before optimization has at least one feature extraction network layer, which includes a first neural network layer and a linear layer. In contrast, the AI ​​model of this application has a corresponding feature extraction network layer, which includes a first neural network layer and a feature dictionary layer. In other words, this application uses the feature dictionary layer to replace the linear layer in the previous AI model. Thus, during the AI ​​model inference process, there is no need to perform computationally intensive linear calculations. Instead, a more computationally efficient dictionary lookup operation is performed, which effectively reduces computational overhead and increases the inference speed of the AI ​​model.

[0015] In one possible implementation, the second feature dictionary includes a first search tree, which is constructed based on multiple input feature and output feature pairs, with the leaf nodes of the first search tree storing the input feature and output feature pairs.

[0016] In other words, the input and output features of the linear layers in the unoptimized AI model are collected, organized, and saved in the form of a search tree to facilitate feature retrieval during the subsequent AI model inference process.

[0017] Optionally, the search tree can be a balanced binary tree to improve the efficiency of the AI ​​model in querying the feature dictionary during the inference process.

[0018] In another possible implementation, the second feature dictionary includes a second search tree, which is constructed based on the clustering results of multiple input feature and output feature pairs. The clustering results include multiple clusters and the corresponding input feature and output feature pairs for each cluster. The leaf nodes of the second search tree store the corresponding input feature and output feature pairs for each cluster.

[0019] In this possible implementation, after collecting multiple input and output feature pairs of the linear layer, clustering is performed on the multiple input and output feature pairs to obtain multiple clusters; a search tree is generated based on the multiple clusters.

[0020] By clustering multiple input and output feature pairs, the number of input and output feature pairs that need to be stored in the feature dictionary is reduced. This increases the query speed of the feature dictionary on the one hand, and reduces the memory overhead of the AI ​​model on the other.

[0021] In another possible implementation, the feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. A specific implementation of this is as follows: based on the first feature vector, a search is performed on the first search tree or the second search tree to obtain the target leaf node of the first search tree or the second search tree. The input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity to the output vector. The output feature in the input feature and output feature pair in the target leaf node is output to obtain the second feature vector.

[0022] By replacing matrix multiplication calculations in the previous linear layer with feature dictionary lookup operations, the computational overhead of the AI ​​model during inference is greatly reduced, the inference efficiency of the AI ​​model is increased, and the output latency of the AI ​​model is reduced.

[0023] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one of the encoders includes a first neural network layer and a feature dictionary layer, wherein the first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

[0024] Compared to existing AI models based on the transformer architecture, the AI ​​model in this application replaces the entire FFN layer in the original AI model with a feature dictionary layer. This eliminates the matrix multiplication calculation of the entire FFN layer during inference, replacing it with a more computationally efficient feature dictionary lookup operation. This significantly reduces the computational overhead of the AI ​​model's inference, improves the inference efficiency of the AI ​​model, reduces the output latency of the AI ​​model, and enhances the user experience.

[0025] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one encoder includes a first neural network layer and a feature dictionary layer. The feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer, and a third feature dictionary sub-layer. The first neural network layer is a scaled dot product attention mechanism layer. The output of the first feature dictionary layer is connected to the input of the scaled dot product attention mechanism layer, the output of the scaled dot product attention mechanism layer is connected to the input of the second feature dictionary layer, and the output of the second feature dictionary layer is connected to the output of the third feature dictionary layer.

[0026] The AI ​​model provided in this application further reduces the computational overhead of the AI ​​model's inference, improves the inference efficiency of the AI ​​model, reduces the output latency of the AI ​​model, and enhances the user experience by replacing the linear layers in the self-attention layer and the FFN layers following the self-attention layer in the transformer-based AI model with feature dictionaries, and replacing the matrix multiplication calculations in the transformer-based AI model with feature dictionary lookup operations.

[0027] Optionally, the AI ​​model provided in this application is a large model, such as a large language model, a large computer vision (CV) model, or a multimodal large model.

[0028] Secondly, this application provides an inference device for an AI model, including an acquisition module and an inference module. The acquisition module is used to acquire input data, which includes at least one of text data, voice data, image data, and video data. The inference module is used to input the input data into the AI ​​model to obtain an inference result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

[0029] In one possible implementation, the feature dictionary layer includes a first feature dictionary, which comprises K dictionary tables. Each of the K dictionary tables includes multiple basis vectors, where K is a positive integer. The inference module is specifically used to: obtain the index value and weight value of each of the K dictionary tables based on the first feature vector; find the target basis vector corresponding to each dictionary table from each dictionary table based on the index value of each dictionary table; and perform a weighted summation of the target basis vectors corresponding to each dictionary table based on the weight value of each dictionary table to obtain the second feature vector.

[0030] In one possible implementation, the feature dictionary layer also includes a query sublayer; a specific implementation of obtaining the index values ​​and weight values ​​of each of the K dictionary tables based on the output vector is as follows: the output vector is used as the input of the query sublayer, and the index values ​​and weight values ​​of each of the K dictionary tables are output.

[0031] Optionally, multiple basis vectors and query sub-layers in each dictionary table are trained based on the training sample set.

[0032] In one possible implementation, the feature dictionary layer includes a second feature dictionary, which is constructed based on multiple input and output feature pairs. The multiple input and output feature pairs are obtained by collecting the input and output features of the linear layer after using the training sample set as input to the AI ​​model before optimization. The AI ​​model before optimization has at least one feature extraction network layer, which includes a first neural network layer and a linear layer.

[0033] In one possible implementation, the second feature dictionary includes a first search tree, which is constructed based on multiple input feature and output feature pairs, with the leaf nodes of the first search tree storing the input feature and output feature pairs.

[0034] In another possible implementation, the second feature dictionary includes a second search tree, which is constructed based on the clustering results of multiple input feature and output feature pairs. The clustering results include multiple clusters and the corresponding input feature and output feature pairs for each cluster. The leaf nodes of the second search tree store the corresponding input feature and output feature pairs for each cluster.

[0035] In another possible implementation, the inference module is specifically used to: search the first search tree or the second search tree based on the first feature vector to obtain the target leaf node of the first search tree or the second search tree, wherein the input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity to the output vector; and output the output feature in the input feature and output feature pair in the target leaf node to obtain the second feature vector.

[0036] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one of the encoders includes a first neural network layer and a feature dictionary layer, wherein the first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

[0037] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one encoder includes a first neural network layer and a feature dictionary layer. The feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer, and a third feature dictionary sub-layer. The first neural network layer is a scaled dot product attention mechanism layer. The output of the first feature dictionary layer is connected to the input of the scaled dot product attention mechanism layer, the output of the scaled dot product attention mechanism layer is connected to the input of the second feature dictionary layer, and the output of the second feature dictionary layer is connected to the output of the third feature dictionary layer.

[0038] Optionally, the AI ​​model provided in this application is a large model, such as a large language model, a large visual model, or a multimodal large model.

[0039] Thirdly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the method described in the first aspect to be implemented.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first aspect to be implemented.

[0041] Fifthly, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the method described in the first aspect.

[0042] In a sixth aspect, embodiments of this application also provide a chip including at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. Attached Figure Description

[0043] Figure 1 shows a schematic diagram of the transformer architecture;

[0044] Figure 2 shows a schematic diagram of an encoder;

[0045] Figure 3 shows a schematic diagram of the encoder structure in the large language model provided by the embodiments of this application;

[0046] Figure 4 is a flowchart illustrating an AI model reasoning method provided in an embodiment of this application;

[0047] Figure 5 illustrates a schematic diagram of a feature dictionary construction process provided in an embodiment of this application;

[0048] Figure 6 shows a schematic diagram of the encoder structure using the large language model provided in the embodiments of this application;

[0049] Figure 7 shows a schematic diagram of the specific structure of the self-attention layer in the encoder;

[0050] Figure 8 shows a schematic diagram of the structure of the self-attention layer in the encoder of the AI ​​model provided in the embodiments of this application;

[0051] Figure 9 shows a schematic diagram of the structure after replacing the feedforward neural network layer in the encoder with the second feature dictionary provided in this application;

[0052] Figure 10 shows a schematic diagram of the encoder using the AI ​​model provided in the embodiments of this application;

[0053] Figure 11 is a schematic diagram of the structure of an AI model inference device provided in an embodiment of this application;

[0054] Figure 12 is a schematic diagram of the structure of the computing device provided in the embodiment of this application. Detailed Implementation

[0055] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0056] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature dictionary" and "second feature dictionary," etc., are used to distinguish different memory chain data, not to describe a specific order of feature dictionaries.

[0057] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0058] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0059] With the iterative upgrades of storage hardware (manifested as larger storage capacity and lower cost per unit of storage), storing more data or model parameters becomes possible. Therefore, designing feature-based storage and query solutions to accelerate LLM has significant practical value and market competitiveness.

[0060] For example, most retrieval enhancement schemes for FFN in Transformer layers construct a learnable parameter table and a hash mapping function (a random fixed function or learnable parameters). While these schemes perform well in shallower LLMs, they perform poorly in deep LLMs with a large number of parameters and in scenarios with large amounts of training data.

[0061] In related technologies, locality-sensitive hashing (LSH) is used to replace linear layers in neural networks to reduce computational cost. First, LSH is constructed to replace the linear transformation computation in FFN, and a strategy is proposed to reduce the update frequency of the hash transformation function in LSH.

[0062] This scheme uses a hash function to replace the computation of the linear layer, but the computation process requires constructing a hash bucket representation based on the hash map. Although the scheme proposes a strategy to reduce the update frequency of the hash function, the function is still continuously updated during training, making the hash representation unsuitable as a fixed feature that can be stored.

[0063] In view of this, this application provides an AI model inference method and apparatus, which uses a feature dictionary containing a large amount of effective information from the training process (such as the input and output features of linear layers) to replace the linear layers in the AI ​​model to optimize the AI ​​model. In the inference process of the optimized AI model, the query operation of the feature dictionary replaces the linear transformation calculation of the linear layer, which significantly reduces the computing power overhead while ensuring the inference accuracy of the AI ​​model, thereby accelerating the inference of the AI ​​model, reducing the output latency of the AI ​​model, and improving the user experience.

[0064] The AI ​​model inference method and apparatus provided in this application can solve the problem of computationally intensive linear calculations, such as a large number of matrix multiplications, performed by the linear layer in the original AI model. Instead, a simpler dictionary lookup operation is performed, and the output of the original linear layer is determined based on the lookup result. While ensuring the accuracy of the AI ​​model, the computational cost is significantly reduced, the inference of the AI ​​model is accelerated, and the output latency of the AI ​​model is reduced.

[0065] The AI ​​model inference method and apparatus provided in this application have a more significant acceleration effect on AI models with larger linear layer parameter scales, especially when applied to various large models. For example, the acceleration effect is particularly significant for large models with large-scale parameters, such as large language models, large visual models, and large multimodal models.

[0066] The following embodiments of this application take the acceleration of inference for large language models as an example to describe in detail the specific implementation of the inference method and device of the AI ​​model provided in this application.

[0067] Figure 1 illustrates a schematic of the Transformer architecture. As shown in Figure 1, the Transformer architecture includes encoder blocks and decoder blocks. An encoder block contains multiple stacked encoders, and a decoder block contains multiple stacked decoders. The input to each encoder is the output of the previous encoder, and the input to each decoder includes the output of the previous decoder and the output of the entire encoder block. The input data is processed by multiple encoders and decoders to obtain the output data. For example, when the Transformer architecture is applied to a text translation model, the input data is "machine learning," and after processing by multiple encoders and decoders in the Transformer architecture, the output is "machine learning."

[0068] Figure 2 shows a schematic diagram of an encoder structure. As shown in Figure 2, the encoder includes a self-attention layer and a feedforward neural network layer. The feedforward neural network layer includes multiple linear layers used to perform linear calculations on the self-attention layer, such as matrix multiplication. Generally, the dimension of the input matrix and the dimension of the output matrix of the feedforward neural network layer are the same.

[0069] The structure of the decoder is similar to that of the encoder, with each decoder also including a self-attention layer and a feedforward neural network layer. As shown in Figure 1, the AI ​​model based on the transformer architecture has multiple encoders and multiple decoders. This means that each encoder and each decoder must perform a matrix multiplication calculation in the feedforward neural network layer. The entire inference process of the AI ​​model involves multiple matrix multiplication calculations in the feedforward neural network layer, which requires significant computational overhead and increases the inference time of the AI ​​model. Large language models based on the transformer architecture have a huge number of parameters (currently reaching hundreds of billions). The linear computation involved in the inference process of large language models is even greater, and it is estimated that the proportion of linear computation in the inference process of large language models may reach more than 50%. Therefore, solving the problem of the large computational time overhead of linear computation in large language models will have a significant acceleration effect on the inference computation time of large language models.

[0070] This application provides an AI model inference method that replaces the feedforward neural network layer in each encoder of a large language model with a feature dictionary layer. The feature dictionary layer includes a feature dictionary that stores the input and output feature information of the feedforward neural network layer. By replacing the linear calculation of the feedforward neural network with the query operation of the feature dictionary, the computationally expensive linear calculation is eliminated in the inference calculation process of the large language model, thereby accelerating the inference of the large language model.

[0071] Figure 3 shows a schematic diagram of the encoder structure in the large language model provided in this application embodiment. As shown in Figure 3, the feedforward neural network layer in the encoder of the large language model is replaced with a feature dictionary. The linear calculation of the original feedforward neural network layer is replaced by the query operation of the feature dictionary, which significantly reduces the computational overhead of the large language model inference process and accelerates the inference of the large language model.

[0072] It should be noted that, taking into account both the hardware performance (e.g., storage size) of the computing device deploying the large language model and the acceleration requirements of the large language model, all feedforward neural network layers in the encoder and decoder of the large language model can be replaced with feature dictionaries, or only some feedforward neural network layers in the encoder and decoder of the large language model can be replaced with feature dictionaries. The more feedforward neural network layers in the large language model that are replaced by feature dictionaries, the better the inference acceleration effect of the large language model, but the corresponding storage overhead of the computing device will also be greater. Conversely, the fewer feedforward neural network layers in the large language model that are replaced by feature dictionaries, the worse the inference acceleration effect of the large language model, but the storage overhead of the computing device will be smaller.

[0073] Figure 4 is a flowchart illustrating an AI model inference method provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application does not specifically limit the specific computing device executing this method; a suitable computing device can be selected as needed. For example, it can be implemented on a terminal device, meaning the AI ​​model provided in this application is deployed on a terminal device to implement the AI ​​model inference method provided in this application. It can also be implemented on both a terminal device and a cloud device (e.g., a cloud-side server), that is, using an edge-cloud collaborative architecture. Alternatively, it can be completed on a cloud device, providing AI model inference services to users in the form of cloud services. For ease of description, the form of the executing entity will not be distinguished in the following text; all will be described as an AI model inference device. As shown in Figure 4, the AI ​​model inference method provided in this application includes at least steps S401 to S404.

[0074] In step S401, input data is acquired.

[0075] Taking an AI model as a large language model, deployed in the cloud, as an example, when a user has a question, they can open a client deployed on their terminal (such as a smart question-and-answer client). The user enters their question in the input box of the interactive interface, such as "Please help me write an advertising slogan for ** company (i.e., the input data can be text data)." The client sends this input data (i.e., the user's question, "Please help me write an advertising slogan for ** company") to the cloud server where the large language model is deployed. The cloud server receives this input data, obtains the user's input data, calls the large language model, and uses this input data as its input. The large language model infers and outputs the corresponding answer based on this input data. For example, the output answer might be "*** company, making data more secure, making the future better."

[0076] In one example, the input data can also be data in other modalities or formats, such as voice data, image data, video data, etc. For example, the input data can be text data, or voice data, or image data, or video data, etc.

[0077] In another example, the input data can also be a mixture of multiple modalities or formats. For example, the input data can be text data + voice data, or text data + image data, or text data + video data, or voice data + image data, or text data + voice data + image data + video data, etc. This application does not specifically limit the modality or format of the input data.

[0078] In step S402, the input data is input into the AI ​​model to obtain the reasoning result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

[0079] This application improves the structure of the AI ​​model by replacing the linear layer in the feature extraction network layer with a feature dictionary layer. During the feature extraction process, the computationally intensive linear calculations are reduced or eliminated, and instead, a more computationally efficient dictionary lookup operation is performed. This significantly reduces computational overhead, accelerates AI model inference, and reduces AI model output latency while maintaining the accuracy of the AI ​​model.

[0080] For example, existing AI models include multi-layer feature extraction network layers. Each feature extraction network layer includes a first neural network layer and a linear layer. The first neural network layer is connected to the linear layer, that is, the output of the first neural network layer is connected to the input of the linear layer. The output of the first neural network layer (i.e., the first feature vector) serves as the input of the linear layer. The linear layer performs linear calculations on the output of the first neural network layer and outputs a feature vector (i.e., the second feature vector).

[0081] For example, existing AI models are large language models based on the transformer architecture (see Figures 1 and 2). These large language models include multiple encoders and decoders. Each encoder includes a self-attention layer and a feedforward neural network layer. The feedforward neural network layer consists of multiple linear layers, with the self-attention layer serving as the first neural network layer. The output of the self-attention layer becomes the input of the feedforward neural network layer, which performs linear calculations on the output of the self-attention layer to output the feature vector encoded by the encoder. We know that the feedforward neural network layer requires computationally intensive linear calculations (such as matrix multiplication). With a large number of parameters in a large language model (including numerous encoders, each with its feedforward neural network layer performing linear calculations), the computational overhead becomes enormous, leading to slower inference speeds, slower output response, and a poor user experience.

[0082] In this embodiment, a feature dictionary that retains the input and output feature information of the feedforward neural network layer can be used to replace the feedforward neural network layer in the existing AI model to obtain a new AI model. When performing inference, this AI model does not need to perform the computationally intensive linear calculations in the feedforward neural network layer. Instead, it can perform a simpler feature dictionary lookup operation, which greatly reduces the inference overhead in the AI ​​model inference process and significantly increases the inference speed of the AI ​​model.

[0083] There are several ways to implement a feature dictionary. The first way is to directly store the input and output feature pairs of the feedforward neural network layer to retain the effective information learned by the large language model from a large training corpus.

[0084] Taking a large language model as an example, the training corpus is used as the input to the large language model. If the large language model in this embodiment replaces the feedforward neural network layer in each encoder of the large language model, then the input feature vector and output feature vector of the feedforward neural network layer in each encoder are collected to obtain the input feature and output feature pair of the feedforward neural network layer in each encoder. For example, if the input of the feedforward neural network layer in a certain encoder of the large language model is feature vector A and the output is feature vector A', then the input feature and output feature pair of the feedforward neural network layer is feature vector A - feature vector A'. Generally speaking, the input feature vector A and the output feature vector A' of the feedforward neural network layer are both multi-dimensional matrices.

[0085] The multiple input and output feature pairs collected for each feedforward neural network layer constitute the input and output feature pair set for each feedforward neural network layer. Based on the input and output feature sets of each feedforward neural network layer, a feature dictionary corresponding to each feedforward neural network layer is generated. This allows the feedforward neural network layer to be replaced in subsequent steps using the feature dictionary corresponding to each feedforward neural network layer, thereby optimizing the large language model and accelerating its inference.

[0086] During the inference process of the large language model, the output vector of the first neural network layer (i.e., the self-attention layer) connected to the feedforward neural network layer is used to query the feature dictionary to obtain the output feature vector that matches the output vector. Here, matching means: using the output vector as the query input feature vector, calculating the similarity between the input feature vector and the input feature vector A in the input feature and output feature pair in the feature dictionary, finding the input feature vector A with the highest similarity, and then identifying the output feature vector A' in the input feature and output feature pair corresponding to input feature vector A as the output feature vector that matches that output vector.

[0087] The second implementation of the feature dictionary is to use learnable parameters to represent the effective information learned by the large language model from a large amount of training corpus. For example, the feature dictionary in this implementation can include multiple dictionary tables, each of which includes multiple basis vectors. The basis vectors in each dictionary table are adjusted by training with multiple input features and output features, so that the output feature vector can be represented by the combination of basis vectors in each dictionary table.

[0088] For example, the feedforward neural network layer in the encoder of a large language model can be replaced with a feature dictionary that requires parameter learning. Then, the large language model can be trained. During the training process, the basis vectors of each dictionary table in the feature dictionary can be adjusted so that the feature dictionary learns to represent the output of the original feedforward neural network layer by combining the basis vectors.

[0089] The generation methods for these two feature dictionaries are described in detail below.

[0090] The first method for generating a feature dictionary involves organizing and storing multiple input and output feature pairs in a format conducive to querying. For example, multiple input and output feature pairs can be organized and stored as a search tree. Specifically, multiple input and output feature pairs can be organized and stored as a balanced binary tree. When performing a feature dictionary query, the balanced binary tree search method is used to quickly find the corresponding query result, i.e., the output feature vector.

[0091] In another example, to further increase the query efficiency of the feature dictionary and reduce its storage overhead, after collecting multiple input and output feature pairs from the feedforward neural network layer, these pairs are clustered and compressed. Then, a balanced binary tree is constructed using the clustered data.

[0092] Figure 5 illustrates a schematic diagram of the feature dictionary construction process provided in an embodiment of this application. As shown in Figure 5, for the construction of the feature dictionary for any feedforward neural network layer, firstly, the training corpus is input into the trained large language model, and then the input feature vector and output feature vector of the feedforward neural network layer are collected to obtain multiple input feature and output feature pairs of the feedforward neural network layer. Then, a preset clustering algorithm is used to cluster the multiple input feature and output feature pairs to obtain multiple clusters. The input feature and output feature pairs at the center of each cluster represent the input feature and output feature pairs in that cluster. Finally, a balanced binary tree is constructed using the input feature and output feature pairs corresponding to the multiple clusters, that is, the input feature and output feature pairs corresponding to the multiple clusters are stored in the leaf nodes of the balanced binary tree according to the construction rules of the balanced binary tree.

[0093] In this embodiment, clustering can be performed on the input feature vectors of multiple input feature and output feature pairs. The clustering result of the input feature vectors represents the clustering result of multiple input feature and output feature pairs. For example, if the input feature and output feature pair is input feature vector A - output feature vector A', when executing the clustering algorithm, cluster analysis is performed on the input feature vector A to obtain multiple clusters. The input feature vector A located at the center of the cluster represents the cluster, resulting in multiple input feature vectors A corresponding to the clusters. This leads to multiple input feature vector A - output feature vector A' pairs corresponding to the clusters. Finally, the multiple input feature vector A - output feature vector A' pairs are stored in the leaf nodes of the balanced binary tree according to the construction rules of the balanced binary tree.

[0094] Clustering can significantly reduce the number of multiple input and output feature pairs, thereby reducing storage overhead and increasing query efficiency.

[0095] It should be noted that the embodiments of this application do not specifically limit which clustering algorithm is used to cluster multiple input and output feature pairs. For example, the clustering algorithm used in the embodiments of this application can be K-means clustering algorithm, hierarchical clustering algorithm, and DBSCAN clustering algorithm, etc., and a suitable clustering algorithm can be selected according to actual needs.

[0096] The second implementation method of the feature dictionary is as follows: the feature dictionary stores learnable parameters, the feature dictionary layer is replaced with the linear layer in the existing AI model, a new AI model is constructed, and then the AI ​​model is trained. During the training process of the AI ​​model, the parameters of the feature dictionary are adjusted.

[0097] For example, we can learn the parameters of the feature dictionary using dictionary learning. We learn multiple basis vectors from each table in the feature dictionary, and then use a weighted sum of these basis vectors to represent the feature vector that matches the output of the first neural network.

[0098] The feature dictionary includes multiple dictionary tables B K Each dictionary table includes multiple learnable basis vectors. The goal of dictionary learning is to learn from multiple dictionary tables B in the feature dictionary. K The basis vectors in the dictionary represent the output feature vectors. The specific dictionary learning process is as follows: the first feature vector X output by the first neural network is used as the input to the feature dictionary. The feature dictionary first uses the input X as the input to the function f(X) (i.e., the query sublayer) to obtain the index values ​​I of each dictionary table. K and the weight C of each dictionary table K Then, using the index value I of each dictionary table K The query retrieves the target basis vectors from each dictionary table. Then utilize the weight value C of each dictionary table K The target basis vectors output by each dictionary table The weighted summation is performed to obtain the output y of the feature dictionary, which is the second feature vector. The second feature vector is then used as the input of the subsequent neural network layer until the output of the AI ​​model is obtained. The output of the AI ​​model is used to adjust the basis vectors and parameters of the function f(X) in each dictionary table of the feature dictionary to obtain the trained feature dictionary.

[0099] This application does not impose specific limitations on the specific dictionary learning algorithm used. Appropriate dictionary learning algorithms can be selected as needed to learn the feature dictionary of this application embodiment.

[0100] By generating a feature dictionary through dictionary learning, and using a few basis vectors to represent the output feature vectors, dimensionality reduction is achieved in a sense, thus accelerating computation. For example, for a set of three-dimensional vectors, we can represent them using a three-dimensional Cartesian coordinate system (three unit vectors x, y, and z). These three vectors can be combined to represent all three-dimensional vectors. This is the learning target of the basis vectors in our feature dictionary. By combining the basis vectors from various dictionary tables, we can represent all the output feature vectors in the input feature and output feature pair.

[0101] It should be noted that the large language model provided in this application embodiment can be obtained by modifying the structure of an existing large language model. For example, the feedforward neural network layer in the encoder of the original large language model can be replaced with a second type of feature dictionary to obtain a large language model with a faster inference speed. When the original large language model is a pre-trained large language model, the parameter-learnable feature dictionary is replaced with the feedforward neural network in the large language model, and then the large language model is trained using training corpus. At this time, other parameters of the large language model are frozen unchanged, and only the parameters of the feature dictionary are adjusted (including the basis vectors and query sub-layers of each dictionary table in the feature dictionary) to obtain a trained feature dictionary. In another example, the original large language model can also be an untrained large language model. In this case, the parameter-learnable feature dictionary is replaced with the feedforward neural network in the large language model, and then the large language model is trained using training corpus. The feature dictionary and the large language model are learned together, that is, the parameters of other layers in the large language model and the parameters of the feature dictionary need to be learned and adjusted during the training process. After the large language model is trained, the trained feature dictionary is obtained.

[0102] The feedforward neural networks in the encoder and / or decoder of the large language model are replaced with the feature dictionary obtained in the preceding steps. In this way, during the inference computation of the large language model, computationally expensive linear calculations (such as matrix multiplication) are eliminated. Instead, feature dictionary lookup operations with lower computational overhead are performed, reducing the computational cost of the entire large language model's inference computation process, accelerating inference speed, and consequently reducing the response output latency of the large language model, thus improving the user experience.

[0103] Figure 6 shows a schematic diagram of the encoder structure using the large language model provided in the embodiments of this application.

[0104] When the feature dictionary is the first type, i.e., a balanced binary tree is used to store the input and output feature pairs, the encoder in the large language model processes as follows: the output of the self-attention layer is used as the input of the feature dictionary; the input vector (i.e., the output vector of the self-attention layer) is used to search the balanced binary tree; based on the similarity between the input vector and the stored features (i.e., the input features in the input and output feature pairs stored in the balanced binary tree), the search proceeds from the root node to the leaf node of the balanced binary tree to determine the target leaf node; the target leaf node outputs the output feature vector from the input and output feature pairs stored thereon, thus obtaining the output feature vector; the output feature vector is used as the input of the next encoder for further processing.

[0105] It should be noted that there are various similarity algorithms for calculating the similarity between the input vector and the stored features, such as Euclidean distance, Manhattan distance, Chebyshev distance, cosine similarity, and Jaccard distance. The appropriate similarity algorithm can be selected according to the actual situation to calculate the similarity between the input vector and the stored features. This application does not make specific limitations on the similarity algorithm used.

[0106] Figure 7 shows a schematic diagram of the self-attention layer in the encoder. As shown in Figure 7, the encoder includes a linear layer, a scaled dot-product attention layer, a feature fusion layer (concat), and a linear layer. It can be seen that the self-attention layer also includes multiple linear layers, which are relatively computationally intensive. Therefore, in order to further increase the inference speed of the large language model, the linear layers in the self-attention layer are also replaced with a feature dictionary.

[0107] Based on the feature dictionary construction algorithm described above, a feature dictionary corresponding to the linear layer in the self-attention layer is constructed. Then, the constructed feature dictionary is used to replace the linear layer in the self-attention layer, thereby further reducing the computational overhead of the large language model and increasing the inference speed of the large language model.

[0108] Figure 8 shows a schematic diagram of the structure of the self-attention layer in the encoder of the AI ​​model provided in the embodiments of this application. As shown in Figure 8, the feature dictionary constructed based on the embodiments of this application replaces the original linear layer in the self-attention layer. During inference, the query operation of the feature dictionary replaces the matrix multiplication matrix calculation of the previous linear layer, reducing the computational overhead and accelerating the inference of the entire large language model.

[0109] Figure 9 shows a schematic diagram of the structure after replacing the feedforward neural network layer in the encoder with the second feature dictionary provided in this application. As shown in Figure 9, by replacing the feedforward neural network in the encoder of the large language model with the feature dictionary constructed in the embodiment of this application, the original calculation process of the feedforward neural network is replaced by a feature dictionary lookup operation, effectively reducing the amount of computation and improving the inference speed of the large language model. The model structure after feature dictionary replacement can be called Dict-LLM. The inference calculation process of the AI ​​model provided in the embodiment of this application is as follows: the output of each encoder after passing through the self-attention layer is used as the input of the feature dictionary. As mentioned above, the feature dictionary includes multiple dictionary tables B. K The specific feature dictionary lookup process is as follows: First, the input X is used as the input to the function f(X), and the index values ​​I of each dictionary table are obtained. K and the weight C of each dictionary table K Then, using the index value I of each dictionary table K The query retrieves the target basis vectors from each dictionary table. Then utilize the weight value C of each dictionary table K The target basis vectors output by each dictionary table The weighted summation yields the output y of the feature dictionary. The specific calculation formula is shown below:

[0110] Figure 10 shows a schematic diagram of the encoder structure of the AI ​​model provided in this application embodiment. As shown in Figure 10, the feature dictionary constructed in this application embodiment replaces all linear layers (including linear layers in the self-attention layer and feedforward neural network layers connected to the self-attention layer) in the encoder of the large language model with the feature dictionary. The inference process of the large language model provided in this application embodiment is as follows: In each encoder, the input X of the encoder is received (the input of the first encoder layer is the output of the embedded representation layer, and the input of the encoder of the subsequent layers is the output of the previous encoder layer), the input X is input to the feature dictionary for querying, and the output vectors Q, K, and V are output, where the expression of the Q vector is Q = DictLLM. Q (X), the expression for the K vector is K = DictLLM K The expression for vector (X), V, is V = DictLLM. V (X). After obtaining the Q vector, K vector, and V vector through the feature dictionary lookup, they are input into the softmax layer for calculation. The calculation result is then used as the input to the feature dictionary, and the output is the original output of the attention layer. This output is then used as the input to the feature dictionary, and the original output feature vector of the feedforward neural network layer is output. This output feature vector is the output of the encoder (i.e., the feature vector output by the encoder).

[0111] Based on the feature vectors output by several feature extraction network layers, determine the inference result of the AI model. Taking the AI model based on the transformer architecture as an example, after being processed by multiple encoder and decoder layers, the feature vectors of the input data are extracted, and then the feature vectors are input into the classifier to output the inference result of the AI model. For example, if the AI model based on the transformer architecture is a translation model and the input data is the Chinese text sequence "machine learning", then multiple encoders and decoders in the AI model process the input data and output the feature vector Y representing the features of the input data "machine learning". Taking this feature vector Y as the input of the classifier, the classifier outputs the probability distribution of each word based on the feature vector Y, and the inference result output based on this probability distribution is the English text sequence "machine learning" corresponding to the Chinese text sequence.

[0112] It should be noted that the inference result is related to the application scenario of the AI model. For example, in the above example, the AI model is a translation model applied to the translation scenario, and the input data is Chinese text data, then the inference result of the AI model is the English translation text corresponding to the Chinese text data. Another example is that if the AI model is a large language model applied to the question-and-answer scenario and the input data is the question text, then the inference result of the AI model is the answer text corresponding to the question text.

[0113] The inference method provided by the embodiments of this application replaces the relatively computationally expensive linear layers in the large language model with feature dictionaries, significantly reducing the computational overhead in the inference process of the large language model, increasing the inference speed of the large language model, reducing the response latency of the large language model, and improving the user experience.

[0114] It should be noted that the first type of feature dictionary stores a relatively large number of features (directly stores the input feature and output feature pairs of the linear layer), so it requires a large storage overhead, but the acceleration effect is good. Through experiments, it is confirmed that using the first type of feature dictionary provided by the embodiments of this application to optimize and accelerate the inference of the large language model, the computational overhead of the query operation of the feature dictionary in the inference process of the large language model is only 1 / 1000 of the computational overhead of the original feedforward neural network, but the output accuracy of the model remains unchanged. Therefore, when using the first type of feature dictionary for inference optimization and acceleration, it is recommended that the storage space of the hardware for deploying the large language model be more than 100TB. Otherwise, it is recommended to use the second type of feature dictionary to optimize and accelerate the large language model. The second type of feature dictionary has a smaller storage overhead for the device. It has been experimentally confirmed that the large language model optimized with the second type of feature dictionary only expands the number of parameters by eight times compared with the original model, but the computational amount is only 1 / 30 of the original model, and the output accuracy of the model remains unchanged.

[0115] For example, the computational cost of the feature dictionary optimized using the first type of feature dictionary (i.e., the feature dictionary constructed from a balanced binary tree) is n*log(N)*d, while the computational cost of the original model's feedforward neural network is n*4d. 2 Where N is the dataset size (1.4T), n is the sentence length (2k), and d is the model feature dimension (10k). Calculations show that the computational cost of the first feature dictionary query scheme is only 1 / 1000 of the original feedforward neural network, achieving a 1000-fold speedup. We conducted experiments on a 1B model, replacing three layers of the feedforward neural network with the feature dictionary, while maintaining the same model output accuracy.

[0116] For example, in response to user input: What should I do if I have a headache?

[0117] The output of the existing large language model is: If your headache is due to a physical ailment, it is recommended that you see a doctor first. If your headache is caused by excessive work stress, lack of sleep, irregular eating habits, etc., then you can try the following methods to relieve your headache:

[0118] 1. Get enough sleep, aiming for 8-9 hours each night. 2. Maintain a healthy diet, avoiding excessive sugar and caffeine intake. 3. Moderate exercise can help relieve stress and anxiety.

[0119] The output of the large language model optimized using the first feature dictionary is as follows: First, consult a doctor or clinic to understand the cause of the pain and possible solutions. Second, you can try the following measures: 1. Rest: Rest relieves pain. 2. Drink water: Drinking plenty of water helps relieve pain. 3. Diet: Avoid greasy, spicy, and irritating foods, and maintain a balanced diet. 4. Medication: If the pain persists, consider using pain medication. 5. Physical therapy: Try physical therapy, such as massage and hot compresses.

[0120] For the large language model optimized with the second feature dictionary, we conducted experiments on five natural language processing tasks, namely PIQA, WinoGande, WSC, ARC-Easy, and ARC-Challenge. Our solution reduces the computational cost by 30 times compared to feedforward neural network layers while keeping the model accuracy almost unchanged.

[0121] The experimental data are shown in the table below:

[0122] Of course, the inference method of the AI ​​model provided in this application embodiment can also be applied to scenarios other than large language models, such as accelerating the inference of convolutional neural networks (CNN).

[0123] For example, a convolutional neural network generally includes four types of layers: convolutional layers, activation function layers (i.e., rectified linear unit, ReLU) layers, pooling layers, and fully connected (FC) layers.

[0124] A convolutional neural network model typically includes multiple alternating convolutional layers, activation function layers, and pooling layers, ultimately feeding data into a fully connected layer to obtain the output. For example, the sequence is: Input Image - Conv-Relu-Conv-Relu-Pooling - Conv-Relu-Conv-Relu-Pooling - Conv-Relu-Conv-Relu-Pooling - Conv-Relu-Conv-Relu-Pooling - FC - Output.

[0125] The function of a convolutional layer is to perform feature recognition on the input image data through multiple filters. Each filter has a scanning range used to scan a certain area of ​​the input image. The calculation result of the current convolutional layer is fed into the next layer (which can be an activation function layer, pooling layer, or fully connected layer) for processing.

[0126] The activation function layer performs a MAX(0,x) operation on the input image data, comparing each value in the input image data with the value 0. If a value is greater than 0, it is retained; otherwise, it is set to 0. The activation function layer provides the sparsity of the input image data (i.e., the percentage of 0 values ​​in the data) without changing the size of the input image data (i.e., the amount of data).

[0127] The function of pooling layers is downsampling, which means extracting data from every other row or column in the two-dimensional matrix of the input data, thereby reducing the size of the input image data.

[0128] The operation of a fully connected layer is similar to that of a convolutional layer, but the difference lies in the fact that the filters in a fully connected layer do not scan a small region of the input image data, but rather scan the entire input image data at once and then output a single value. A fully connected layer contains multiple filters, each corresponding to a different, very specific image feature. The output value is equivalent to a "score," representing the "probability" that the input image data contains these features.

[0129] Typically, the computational cost of convolutional and fully connected layers in a convolutional neural network (CNN) model can account for over 90% of the total computational cost. Convolutional layers are essentially a variant of linear layers and still fall under the category of linear layers, while fully connected layers consist of multiple linear layers. Therefore, the method provided in this application can be used to optimize existing CNN models. This involves replacing the convolutional layers in the CNN model with a feature dictionary that retains a large amount of effective information from the training process (such as the input and output features of the convolutional layers, or learnable parameters, such as basis vectors in multiple dictionary tables). In this optimized CNN model, the feature dictionary lookup operation replaces the convolution operation during inference, significantly reducing computational overhead while maintaining the inference accuracy of the CNN model. This accelerates inference, reduces output latency, and improves the user experience.

[0130] In another example, to further accelerate the inference of the convolutional neural network model, the feature dictionary layer of this application is used to replace all convolutional layers and fully connected layers in the convolutional neural network model. In this way, during the inference process of the convolutional neural network model, the more computationally efficient feature dictionary query operation replaces the more computationally efficient convolutional layer convolution operation and fully connected layer linear calculation operation, significantly reducing computational overhead and accelerating the inference of the convolutional neural network model.

[0131] The detailed implementation of the feature dictionary can be found in the description above, and will not be repeated here for the sake of brevity.

[0132] Based on the same concept as the aforementioned embodiment of the AI ​​model inference method, this application also provides an AI model inference device 1100, which can be deployed in a terminal device or a cloud server to improve the inference speed of the AI ​​model. The AI ​​model inference device 1100 includes units or modules for implementing the various steps in the AI ​​model inference method shown in Figures 4-10.

[0133] Figure 11 is a schematic diagram of the structure of an AI model inference device provided in an embodiment of this application. As shown in Figure 11, the AI ​​model inference device 1100 includes at least an acquisition module 1101, an input module 1102, and an inference module 1103. The acquisition module 1101 is used to acquire input data, which includes at least one of text data, voice data, image data, and video data. The inference module 1102 is used to input the input data into the AI ​​model to obtain an inference result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

[0134] In one possible implementation, the feature dictionary layer includes a first feature dictionary, which includes K dictionary tables. Each of the K dictionary tables includes multiple basis vectors, where K is a positive integer. The inference module 1103 is specifically used to: obtain the index value and weight value of each dictionary table in the K dictionary tables based on the first feature vector; find the target basis vector corresponding to each dictionary table from each dictionary table based on the index value of each dictionary table; and perform a weighted summation of the target basis vectors corresponding to each dictionary table based on the weight value of each dictionary table to obtain the second feature vector.

[0135] In one possible implementation, the feature dictionary layer also includes a query sublayer; a specific implementation of obtaining the index values ​​and weight values ​​of each of the K dictionary tables based on the output vector is as follows: the output vector is used as the input of the query sublayer, and the index values ​​and weight values ​​of each of the K dictionary tables are output.

[0136] Optionally, multiple basis vectors and query sub-layers in each dictionary table are trained based on the training sample set.

[0137] In one possible implementation, the feature dictionary layer includes a second feature dictionary, which is constructed based on multiple input and output feature pairs. These multiple input and output feature pairs are obtained by collecting the input and output features of the linear layer after using the training sample set as input to the AI ​​model before optimization. The AI ​​model before optimization has at least one feature extraction network layer, which includes a first neural network layer and a linear layer.

[0138] In one possible implementation, the second feature dictionary includes a first search tree, which is constructed based on multiple input feature and output feature pairs, with the leaf nodes of the first search tree storing the input feature and output feature pairs.

[0139] In another possible implementation, the second feature dictionary includes a second search tree, which is constructed based on the clustering results of multiple input feature and output feature pairs. The clustering results include multiple clusters and the corresponding input feature and output feature pairs for each cluster. The leaf nodes of the second search tree store the corresponding input feature and output feature pairs for each cluster.

[0140] In another possible implementation, the inference module 1103 is specifically used to: search the first search tree or the second search tree based on the first feature vector to obtain the target leaf node of the first search tree or the second search tree, wherein the input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity to the output vector; and output the output feature in the input feature and output feature pair in the target leaf node to obtain the second feature vector.

[0141] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one of the encoders includes a first neural network layer and a feature dictionary layer, wherein the first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

[0142] In another possible implementation, the AI ​​model is a transformer-based AI model, with several feature extraction network layers including several encoders. At least one encoder includes a first neural network layer and a feature dictionary layer. The feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer, and a third feature dictionary sub-layer. The first neural network layer is a scaled dot product attention mechanism layer. The output of the first feature dictionary layer is connected to the input of the scaled dot product attention mechanism layer, the output of the scaled dot product attention mechanism layer is connected to the input of the second feature dictionary layer, and the output of the second feature dictionary layer is connected to the output of the third feature dictionary layer.

[0143] Optionally, the AI ​​model provided in this application embodiment is a large model, such as a large language model, a large visual model, or a multimodal large model.

[0144] The AI ​​model inference device 1100 according to the embodiments of this application can correspond to the execution of the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the AI ​​model inference device 1100 are respectively to implement the corresponding processes of each method in FIG4-10. For the sake of brevity, they will not be described again here.

[0145] This application also provides a computing device including at least one processor, a memory, and a communication interface, wherein the processor is used to execute the method described in FIG4-10.

[0146] Figure 12 is a schematic diagram of the structure of the computing device provided in the embodiment of this application.

[0147] As shown in Figure 12, the computing device 1200 includes at least one processor 1201, a memory 1202, and a communication interface 1203. The processor 1201, memory 1202, and communication interface 1203 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1203 is used to send and / or receive data from other devices. The memory 1202 stores computer instructions, which the processor 1201 executes to perform the methods described in the aforementioned method embodiments. This accelerates the inference of the AI ​​model, significantly reduces the computational overhead during inference while maintaining model accuracy, thereby accelerating AI model inference and reducing output latency.

[0148] It should be understood that, in the embodiments of this application, the processor 1201 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0149] The memory 1202 may include read-only memory and random access memory, and provides instructions and data to the processor 1201. The memory 1202 may also include non-volatile random access memory.

[0150] The memory 1202 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0151] It should be understood that the computing device 1200 according to the embodiments of this application can execute the method shown in Figures 4-10 of the embodiments of this application. For a detailed description of the implementation of the method, please refer to the above text. For the sake of brevity, it will not be repeated here.

[0152] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0153] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0154] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0155] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0157] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A reasoning method for an AI model, characterized in that, include: Acquire input data, which includes at least one of text data, voice data, image data, and video data; The input data is input into an AI model to obtain a reasoning result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

2. The method according to claim 1, characterized in that, The feature dictionary layer includes a first feature dictionary, which includes K dictionary tables. Each of the K dictionary tables includes multiple basis vectors, where K is a positive integer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector, including: Based on the first feature vector, the index value and weight value of each dictionary table in the K dictionary tables are obtained; The target basis vector corresponding to each dictionary table is obtained by searching the dictionary table based on the index value of each dictionary table; The second feature vector is obtained by weighting and summing the target basis vectors corresponding to each dictionary table based on the weight values ​​of each dictionary table.

3. The method according to claim 2, characterized in that, The feature dictionary layer also includes a query sublayer; The process of obtaining the index value and weight value of each of the K dictionary tables based on the output vector includes: The output vector is used as the input to the query sub-layer, and the index value and weight value of each of the K dictionary tables are output.

4. The method according to claim 3, characterized in that, The multiple basis vectors in each dictionary table and the query sub-layer are obtained based on the training sample set.

5. The method according to any one of claims 1-4, characterized in that, The feature dictionary layer includes a second feature dictionary, which is constructed based on multiple input and output feature pairs. The multiple input and output feature pairs are obtained by collecting the input and output features of the linear layer after using the training sample set as the input of the AI ​​model before optimization. The at least one feature extraction network layer of the AI ​​model before optimization includes the first neural network layer and the linear layer.

6. The method according to claim 5, characterized in that, The second feature dictionary includes a first search tree, which is constructed based on the plurality of input feature and output feature pairs, and the leaf nodes of the first search tree store the input feature and output feature pairs.

7. The method according to claim 5, characterized in that, The second feature dictionary includes a second search tree, which is constructed based on the clustering results of the multiple input feature and output feature pairs. The clustering results include multiple clusters and the input feature and output feature pairs corresponding to each cluster. The leaf nodes of the second search tree store the input feature and output feature pairs corresponding to each cluster.

8. The method according to claim 6 or 7, characterized in that, The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector, including: Based on the first feature vector, a search is performed on the first search tree or the second search tree to obtain the target leaf node of the first search tree or the second search tree. The input feature in the input feature and output feature pair stored in the target leaf node has the highest similarity to the output vector. The output feature of the input feature and output feature pair in the target leaf node is output to obtain the second feature vector.

9. The method according to any one of claims 1-8, characterized in that, The AI ​​model is an AI model based on the transformer architecture. The plurality of feature extraction network layers include a plurality of encoders. At least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer. The first neural network layer is a self-attention layer, and the self-attention layer is connected to the feature dictionary layer.

10. The method according to any one of claims 1-8, characterized in that, The AI ​​model is an AI model based on the transformer architecture. The plurality of feature extraction network layers include a plurality of encoders. At least one of the plurality of encoders includes the first neural network layer and the feature dictionary layer. The feature dictionary layer includes a first feature dictionary sub-layer, a second feature dictionary sub-layer and a third feature dictionary sub-layer. The first neural network layer is a scaled dot product attention mechanism layer. The output of the first feature dictionary layer is connected to the input of the scaled dot product attention mechanism layer, the output of the scaled dot product attention mechanism layer is connected to the input of the second feature dictionary layer, and the output of the second feature dictionary layer is connected to the output of the third feature dictionary layer.

11. The method according to any one of claims 1-10, characterized in that, The AI ​​model includes any one of the following: large language model, visual large language model, and multimodal large model.

12. An inference device for an AI model, characterized in that, include: An acquisition module is used to acquire input data, which includes at least one of text data, voice data, image data, and video data; The inference module is used to input the input data into the AI ​​model to obtain the inference result. The AI ​​model includes several feature extraction network layers. At least one of the feature extraction network layers includes a first neural network layer and a feature dictionary layer. The first neural network layer is connected to the feature dictionary layer. The feature dictionary layer is used to perform a dictionary lookup on the first feature vector output by the first neural network layer to obtain a second feature vector that matches the first feature vector. The second feature vector is used as the output vector of the feature extraction network layer to which the first neural network layer belongs.

13. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-11 to be implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-11 to be implemented.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the method as described in any one of claims 1-11 to be implemented.

Images