Model inference method, electronic device, storage medium, and computer program product

Abstract

The application provides a model reasoning method, an electronic device, a storage medium and a computer program product, the method comprising: obtaining second vectors output by target expert modules in a current layer based on the first vector of the current layer, the target expert modules being determined from a plurality of expert modules in the current layer based on the first vector of the current layer; determining target block vectors corresponding to the second vectors from block vectors based on the similarity between the second vectors of any target expert module and the block vectors, the block vectors being determined after the key-value matrix is cut; inputting the second vectors to a next layer, loading the key-value matrix of the target block vectors corresponding to the second vectors, and performing model reasoning to obtain third vectors output by the next layer. The model reasoning method, the electronic device, the storage medium and the computer program product provided by the application can increase the accuracy of the reasoning result on the basis of accelerating the reasoning speed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model reasoning method, electronic device, storage medium, and computer program product. Background Technology

[0002] In the inference process of Large Language Models (LLMs), new tokens need to be generated based on historical token information. This is especially true when performing long token inference, which requires storing a large amount of historical token information. In addition, the inference process requires loading all key-value matrices (KVcache) corresponding to historical token information, which leads to slow inference speed. Summary of the Invention

[0003] This invention provides a model reasoning method, electronic device, storage medium, and computer program product to address the drawback of slow reasoning speed in large language models.

[0004] This invention provides a model reasoning method, comprising:

[0005] Based on each target expert module in the current layer, the first vector of the current layer is applied to obtain the second vector output by each target expert module. Each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer.

[0006] Based on the similarity between the second vector of any target expert module and each block vector, the target block vector corresponding to each second vector is determined from each block vector; each block vector is determined after dividing the key value matrix into blocks.

[0007] Each second vector is input to the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for model inference to obtain the third vector output by the next layer.

[0008] According to a model reasoning method provided by the present invention, each target expert module is determined based on the following steps:

[0009] The first vector is input to the attention module of the current layer, and the attention module of the current layer performs attention calculation to determine the attention calculation result of the current layer.

[0010] The routing result is obtained by performing routing calculation on the attention calculation result of the current layer;

[0011] Based on the routing results, each target expert module is determined from the plurality of expert modules.

[0012] According to a model reasoning method provided by the present invention, the step of obtaining a second vector output by each target expert module based on the first vector of the current layer and applying the first vector of the current layer includes:

[0013] The attention calculation results of the current layer are summed, and the sum is input to each target expert module to obtain the second vector output by each target expert module.

[0014] According to a model reasoning method provided by the present invention, the step of determining the target block vector corresponding to each second vector from each block vector based on the similarity between the second vector of any target expert module and each block vector includes:

[0015] Sort the vector blocks according to their similarity from high to low.

[0016] The pre-defined number of block vectors that are sorted first are used as the target block vectors of the corresponding second vector.

[0017] According to a model inference method provided by the present invention, each block vector is determined based on the following steps:

[0018] The key-value matrix is ​​divided into multiple block matrices along the sequence dimension.

[0019] Average pooling is performed on the vectors within each segmentation matrix to obtain the corresponding segmented vectors.

[0020] According to a model inference method provided by the present invention, the step of inputting each second vector to the next layer and loading the key-value matrix of the target block vector corresponding to each second vector for model inference to obtain the third vector output by the next layer includes:

[0021] Each second vector is input into the attention module of the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for attention calculation to obtain the attention calculation result corresponding to each second vector;

[0022] The third vector is obtained based on the attention calculation results corresponding to each second vector.

[0023] According to a model inference method provided by the present invention, obtaining the third vector based on the attention calculation results corresponding to each second vector includes:

[0024] The attention calculation results corresponding to each second vector are added together or weighted and fused to obtain the third vector.

[0025] The present invention also provides a model reasoning device, comprising:

[0026] The first inference unit is used to obtain the second vector output by each target expert module based on the first vector of the current layer. Each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer.

[0027] The vector determination unit is used to determine the target block vector corresponding to each second vector from each block vector based on the similarity between the second vector of any target expert module and each block vector; each block vector is determined after dividing the key value matrix into blocks;

[0028] The second inference unit is used to input each second vector into the next layer, and load the key value matrix of the target block vector corresponding to each second vector to perform model inference, so as to obtain the third vector output by the next layer.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model inference method as described above.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model inference method as described above.

[0031] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model inference method as described above.

[0032] The model inference method, electronic device, storage medium, and computer program product provided by this invention do not need to load all KVcache during model inference at the next layer; instead, they only load a portion of the KVcache, thereby improving the inference speed. Furthermore, based on the similarity between the second vector of each target expert module and the vectors of each block, this invention selects the key-value matrix corresponding to each target expert module, enabling different target expert modules to process different KVcaches, thus improving the accuracy of the inference results. In other words, this invention increases the accuracy of inference results while accelerating inference speed. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1This is a flowchart illustrating the model reasoning method provided by the present invention;

[0035] Figure 2 This is a schematic diagram of the model structure provided by the present invention;

[0036] Figure 3 This is a schematic diagram of the key matrix block provided by the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of the model reasoning device provided by the present invention;

[0038] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] When performing inference in a large language model, new tokens need to be generated based on historical token information. When performing inference with long tokens, a large amount of historical token information needs to be stored. Loading all the KVcache corresponding to the historical token information would slow down the inference speed. Therefore, related technologies improve the inference speed of the model by loading part of the KVcache during each inference. The specific methods are as follows:

[0041] The relevant technology first divides the KVcache into blocks, performs average pooling on the tokens in each block, and finally obtains the vector representation of each block. The vector representation of each block is then compared with the output token vector, and k sets of vector representations with high similarity are selected as the target block vector. In the next inference, only the KVcache corresponding to the target block vector is loaded, thereby achieving the effect of accelerating inference.

[0042] Furthermore, some related technologies employ a Mix of Experts (MOE) structure within a large language model to load a portion of the KVcache for inference. After each expert module (the FFN layer of the large language model includes multiple FFNs, each representing an expert module, with different expert modules having varying sensitivities to different semantics) completes its computation, the results are summed, and the sum is used to calculate the output of the current layer. The next layer, when performing attention calculations, selects and loads the corresponding KVcache based on the current layer's output. However, in this method, all FFNs load the same KVcache information, affecting the accuracy of the inference results. Here, FFN refers to a Feed-Forward Network.

[0043] In response, this invention provides a model reasoning method. Figure 1 This is a flowchart illustrating the model reasoning method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0044] Step 110: Based on each target expert module in the current layer, apply the first vector of the current layer to obtain the second vector output by each target expert module. Each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer.

[0045] Here, the first vector can be understood as the output vector of the previous layer. The current layer includes multiple expert modules, each with varying sensitivities to different semantics, meaning they differ in importance when handling different tasks. The target expert modules in the current layer can be understood as those with higher importance in handling the corresponding task of the current layer. Each target expert module is determined from the multiple expert modules in the current layer based on the first vector of the current layer. These expert modules can be constructed based on FFN (Freeform Array Numerator).

[0046] Optionally, a weight can be assigned to each expert module based on the first vector. The larger the weight, the more important the corresponding expert module is in the corresponding task. For example, the weights of each expert module can be sorted in descending order, and the expert modules corresponding to the top k weights can be selected as the target expert modules, where k≥2.

[0047] After determining the target expert module, the target expert module determines the corresponding second vector based on the first vector of the current layer. Since different target expert modules have different sensitivities to different semantics, they can process different semantic information in the first vector. In other words, each target expert module focuses on its area of ​​expertise, thereby improving the accuracy of the final inference result.

[0048] Step 120: Based on the similarity between the second vector of any target expert module and each block vector, determine the target block vector corresponding to each second vector from each block vector; each block vector is determined after dividing the key value matrix into blocks.

[0049] Specifically, the similarity between the second vector and any block vector is used to characterize the importance of the corresponding block vector in the task corresponding to the second vector. The higher the similarity, the higher the importance of the corresponding block vector. Therefore, embodiments of the present invention can use block vectors with high similarity to the second vector as target block vectors. For example, the m block vectors with the highest similarity can be used as target block vectors corresponding to the second vector.

[0050] Optionally, the similarity between the second vector of any target expert module and each block vector can be determined based on the distance between the second vector of any target expert module and each block vector. The shorter the distance, the higher the similarity.

[0051] Step 130: Input each second vector into the next layer, and load the key value matrix of the target block vector corresponding to each second vector for model inference to obtain the third vector output by the next layer.

[0052] Specifically, when performing model inference at the next layer, it is not necessary to load all the KVcache, but only a portion of the KVcache (i.e., the KVcache corresponding to the target block vector), which can improve the inference speed of the model.

[0053] Furthermore, different target expert modules have varying sensitivities to different semantics, resulting in different target block vectors corresponding to the second vectors of different target expert modules, and consequently, different loaded key-value matrices (KVcache). In other words, this embodiment of the invention utilizes the characteristic that each target expert module has different sensitivities to different semantics to obtain the output results (second vectors) of each target expert module, and selects the corresponding key-value matrices for each target expert module based on the output results, thereby enabling different target expert modules to process different KVcaches, thus improving the accuracy of the inference results.

[0054] The model inference method provided in this invention does not require loading all KVcache during model inference at the next layer; instead, it loads only a portion of the KVcache, thereby improving the inference speed. Furthermore, this invention selects the key-value matrix corresponding to each target expert module based on the similarity between the second vector of each target expert module and the block vectors, enabling different target expert modules to process different KVcaches, thus improving the accuracy of the inference results. In other words, this invention increases the accuracy of inference results while accelerating inference speed.

[0055] As an optional embodiment, the model here can be a text generation model. The first vector can be determined based on the input text, and the output text corresponding to the input text can be obtained by using the model inference method described above. This can not only improve the efficiency of text generation, but also improve the accuracy of text generation.

[0056] Based on the above embodiments, each target expert module is determined according to the following steps:

[0057] The first vector is input into the attention module of the current layer, which performs attention calculations and determines the attention calculation result of the current layer.

[0058] Perform routing calculations on the attention calculation results of the current layer to obtain the routing results;

[0059] The target expert module is determined from multiple expert modules based on the routing results.

[0060] Specifically, the attention module of the current layer is used to perform attention calculation, obtain the attention calculation result of the current layer, and perform routing calculation on the attention calculation result of the current layer to obtain the routing result. The routing result indicates the weight of each expert module. The larger the weight, the higher the importance of the corresponding expert module in the corresponding task.

[0061] For example, the weights of each expert module can be sorted in descending order, and the expert modules corresponding to the top k weights can be selected as the target expert modules, where k≥2.

[0062] Based on any of the above embodiments, based on each target expert module of the current layer, the first vector of the current layer is applied to obtain the second vector output by each target expert module, including:

[0063] The attention calculation results of the current layer are summed, and the sum is input into each target expert module to obtain the second vector output by each target expert module.

[0064] Specifically, the attention calculation results of each layer in the current layer are summed to obtain a summed result. Each target expert module then processes this summed result to obtain a second vector. In other words, each target expert module in the current layer outputs its corresponding second vector, which is then used as input to the next layer. The next layer can then select the appropriate KVcache for each target expert module based on its second vector, allowing different target expert modules to process different KVcaches and improving the accuracy of the inference results.

[0065] Based on any of the above embodiments, determining the target block vector corresponding to each second vector from each block vector based on the similarity between the second vector of any target expert module and each block vector includes:

[0066] Sort the vector blocks according to their similarity from high to low.

[0067] The pre-defined number of block vectors that are sorted first are used as the target block vectors of the corresponding second vector.

[0068] Specifically, the similarity between the second vector and any block vector is used to characterize the importance of the corresponding block vector in the task corresponding to the second vector. The higher the similarity, the higher the importance of the corresponding block vector. The similarity between the second vector of any target expert module and each block vector can be measured using cosine similarity, Euclidean distance, etc. For example, the lower the cosine similarity, the higher the similarity; the smaller the Euclidean distance, the higher the similarity.

[0069] Based on the calculated similarity, all segmented vectors are sorted in descending order of similarity. A preset number of segmented vectors with the highest similarity are selected from the sorted list as the target segmented vector for the corresponding second vector. The preset number can be set according to actual needs, and this embodiment of the invention does not impose a specific limitation on it.

[0070] Based on any of the above embodiments, each block vector is determined based on the following steps:

[0071] In the sequence dimension, the key-value matrix is ​​divided into blocks to obtain multiple block matrices;

[0072] Average pooling is performed on the vectors within each segmentation matrix to obtain the corresponding segmented vectors.

[0073] Here, the key-value matrix can be understood as all KVcache. The key-value matrix is ​​divided along the sequence dimension (i.e., the seq dimension) to obtain multiple block matrices. The block size can be determined according to actual needs, and this embodiment of the invention does not impose a specific limitation on it.

[0074] For each segmentation matrix, there are multiple vectors in the seq dimension. The average pooling operation is performed on the above multiple vectors to obtain a hidden layer dimension vector (hidden_dim vector), which is the corresponding segmentation vector.

[0075] Based on any of the above embodiments, each second vector is input to the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for model inference to obtain the third vector output by the next layer, including:

[0076] Each second vector is input into the attention module of the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for attention calculation to obtain the attention calculation result corresponding to each second vector;

[0077] The third vector is obtained based on the attention calculation results corresponding to each second vector.

[0078] Specifically, the second vector output by each target expert module is used as input and passed to the attention module of the next layer. At the same time, for each second vector, the attention module of the next layer loads the key-value matrix of the target block vector corresponding to each second vector and performs attention calculations respectively to obtain the attention calculation results corresponding to each second vector, that is, to obtain multiple attention calculation results.

[0079] After obtaining the attention calculation results corresponding to each second vector, the attention calculation results corresponding to each second vector can be added or weighted and fused to obtain the third vector. Wherein, if the current layer is layer n and the next layer is layer n+1, then layer n+1 determines the routing result based on the third vector, selects the target expert module from multiple expert modules in layer n+1 based on the routing result, and repeats the steps of the above embodiment for model inference until the last layer outputs the corresponding token.

[0080] Figure 2 This is a schematic diagram of the model structure provided by the present invention, such as... Figure 2 As shown, the current layer (1MOE Layer) includes attention module Attn1, expert modules FFN0, FFN1, FFN2 and FFN3, and the next layer (2MOE Layer) includes attention module Attn2, expert modules FFN0, FFN1, FFN2 and FFN3.

[0081] Based on any of the above embodiments, the present invention also provides a model reasoning method, which is described below. Figure 2 The model structure shown is used as an example to illustrate the model's reasoning method, which specifically includes:

[0082] The first vector h is input into the attention module Attn1 to obtain the routing result 1. Based on the routing result, FFN0 and FFN2 are determined as the target expert modules from FFN0, FFN1, FFN2 and FFN3. The key-value matrix KVcache1 to be loaded is determined based on the second vector output by FFN0, and the key-value matrix KVcache2 to be loaded is determined based on the second vector output by FFN2.

[0083] KVcache1 and KVcache2 are determined based on the following steps:

[0084] In the seq dimension, the key-value matrix is ​​divided into blocks to obtain multiple block matrices;

[0085] Perform average pooling on the vectors within each block matrix to obtain the corresponding block vectors;

[0086] Based on the similarity between the second vector output by FFN0 and each block vector, the target block vector corresponding to FFN0 is determined, and the KVcache corresponding to the target block vector is taken as KVcache1;

[0087] Based on the similarity between the second vector output by FFN2 and each block vector, the target block vector corresponding to FFN2 is determined, and the KVcache corresponding to the target block vector is taken as KVcache2.

[0088] Figure 3 This is a schematic diagram of the key matrix block provided by the present invention, such as... Figure 3 As shown, the key matrix (Kcache) contains 640 tokens, i.e., the sequence number (seq) is 640. First, the Kcache is divided along the seq dimension, with each chunk having a seq of 128. Taking chunk0 as an example, the 128 vectors within chunk0 undergo an average pooling operation to obtain a hidden_dim vector. This vector is the chunk vector 0 corresponding to chunk0. Similarly, we can obtain the chunk vector 1 corresponding to chunk1, the chunk vector 2 corresponding to chunk2, the chunk vector 3 corresponding to chunk3, and the chunk vector 4 corresponding to chunk4.

[0089] The similarity of block vectors 0, 1, 2, 3, and 4 with the second vector output by FFN0 is compared (e.g., relative distance calculation is performed). The two block vectors with the highest similarity to the second vector output by FFN0 are determined to be block vector 0 and block vector 4. Then, the Kcache corresponding to block vector 0 and block vector 4 is loaded for the next calculation.

[0090] The block vectors 0, 1, 2, 3, and 4 are compared with the second vector output by FFN2 (e.g., relative distance calculation is performed). The two block vectors with the highest similarity to the second vector output by FFN2 are determined to be block vector 1 and block vector 2. Then, the Kcache corresponding to block vector 1 and block vector 2 is loaded for the next calculation.

[0091] After determining KVcache1 and KVcache2, the second vector output by FFN0 is input into the attention module Attn2 in the 2MOE Layer, and KVcache1 is loaded for calculation to obtain attention calculation result 1. Simultaneously, the second vector output by FFN2 is input into the attention module Attn2 in the 2MOE Layer, and KVcache2 is loaded for calculation to obtain attention calculation result 2. Attention calculation result 1 and attention calculation result 2 are then mixed (e.g., added or weighted fusion) to obtain the third vector.

[0092] Similarly, the 2MOE Layer determines the routing result based on the third vector, and selects the target expert module from FFN0, FFN1, FFN2 and FFN3 in 2MOELayer based on the routing result, and executes the steps of the above embodiment until the attention calculation result of the last layer is mixed and the token is output when inference reaches the last layer.

[0093] The model reasoning apparatus provided by the present invention is described below. The model reasoning apparatus described below and the model reasoning method described above can be referred to in correspondence.

[0094] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the model inference device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0095] The first reasoning unit 410 is used to obtain the second vector output by each target expert module based on the first vector of the current layer and by applying the first vector of the current layer. Each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer.

[0096] The vector determination unit 420 is used to determine the target block vector corresponding to each second vector from each block vector based on the similarity between the second vector of any target expert module and each block vector; each block vector is determined after dividing the key value matrix into blocks;

[0097] The second inference unit 430 is used to input each second vector into the next layer, and load the key value matrix of the target block vector corresponding to each second vector to perform model inference, so as to obtain the third vector output by the next layer.

[0098] Based on any of the above embodiments, each target expert module is determined based on the following steps:

[0099] The first vector is input into the attention module of the current layer, which performs attention calculations and determines the attention calculation result of the current layer.

[0100] Perform routing calculations on the attention calculation results of the current layer to obtain the routing results;

[0101] The target expert module is determined from multiple expert modules based on the routing results.

[0102] Based on any of the above embodiments, based on each target expert module of the current layer, the first vector of the current layer is applied to obtain the second vector output by each target expert module, including:

[0103] The attention calculation results of the current layer are summed, and the sum is input into each target expert module to obtain the second vector output by each target expert module.

[0104] Based on any of the above embodiments, determining the target block vector corresponding to each second vector from each block vector based on the similarity between the second vector of any target expert module and each block vector includes:

[0105] Sort the vector blocks according to their similarity from high to low.

[0106] The pre-defined number of block vectors that are sorted first are used as the target block vectors of the corresponding second vector.

[0107] Based on any of the above embodiments, each block vector is determined based on the following steps:

[0108] In the sequence dimension, the key-value matrix is ​​divided into blocks to obtain multiple block matrices;

[0109] Average pooling is performed on the vectors within each segmentation matrix to obtain the corresponding segmented vectors.

[0110] Based on any of the above embodiments, each second vector is input to the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for model inference to obtain the third vector output by the next layer, including:

[0111] Each second vector is input into the attention module of the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for attention calculation to obtain the attention calculation result corresponding to each second vector;

[0112] The third vector is obtained based on the attention calculation results corresponding to each second vector.

[0113] Based on any of the above embodiments, a third vector is obtained based on the attention calculation results corresponding to each second vector, including:

[0114] The attention calculation results corresponding to each second vector are added together or weighted and fused to obtain the third vector.

[0115] Figure 5This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include a processor 510, a memory 520, a communication interface 530, and a communication bus 540, wherein the processor 510, memory 520, and communication interface 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 520 to execute a model inference method, which includes: based on each target expert module in the current layer, applying the first vector of the current layer to obtain a second vector output by each target expert module, wherein each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer; based on the similarity between the second vector of any target expert module and each block vector, determining the target block vector corresponding to each second vector from each block vector; each block vector is determined after dividing the key value matrix into blocks; inputting each second vector to the next layer, and loading the key value matrix of the target block vector corresponding to each second vector for model inference to obtain the third vector output by the next layer.

[0116] Furthermore, the logical instructions in the aforementioned memory 520 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the model inference method provided by the above methods, the method comprising: based on each target expert module of the current layer, applying a first vector of the current layer to obtain a second vector output by each target expert module, wherein each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer; based on the similarity between the second vector of any target expert module and each block vector, determining the target block vector corresponding to each second vector from each block vector; each block vector is determined after dividing the key value matrix into blocks; inputting each second vector to the next layer, and loading the key value matrix of the target block vector corresponding to each second vector for model inference to obtain a third vector output by the next layer.

[0118] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the aforementioned model inference methods. The method includes: based on each target expert module of the current layer, applying a first vector of the current layer to obtain a second vector output by each target expert module, wherein each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer; based on the similarity between the second vector of any target expert module and each block vector, determining the target block vector corresponding to each second vector from each block vector; each block vector is determined after dividing a key-value matrix into blocks; inputting each second vector to the next layer, and loading the key-value matrix of the target block vector corresponding to each second vector for model inference to obtain a third vector output by the next layer.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model reasoning method, characterized in that, include: Based on each target expert module in the current layer, a second vector is obtained by applying the first vector of the current layer. Each target expert module is determined from multiple expert modules in the current layer based on the first vector of the current layer. The first vector is determined based on the input text. Based on the similarity between the second vector of any target expert module and each block vector, the target block vector corresponding to each second vector is determined from each block vector; Each block vector is determined by dividing the key-value matrix into blocks; Each second vector is input to the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for model inference to obtain the third vector output by the next layer.

2. The model reasoning method according to claim 1, characterized in that, Each target expert module is determined based on the following steps: The first vector is input to the attention module of the current layer, and the attention module of the current layer performs attention calculation to determine the attention calculation result of the current layer. The routing result is obtained by performing routing calculation on the attention calculation result of the current layer; Based on the routing results, each target expert module is determined from the plurality of expert modules.

3. The model reasoning method according to claim 2, characterized in that, The second vector output by each target expert module based on the current layer is obtained by applying the first vector of the current layer, including: The attention calculation results of the current layer are summed, and the sum is input to each target expert module to obtain the second vector output by each target expert module.

4. The model reasoning method according to any one of claims 1 to 3, characterized in that, The similarity between the second vector based on any target expert module and each block vector, and the determination of the target block vector corresponding to each second vector from each block vector, includes: Sort the vector blocks according to their similarity from high to low. The pre-defined number of block vectors that are sorted first are used as the target block vectors of the corresponding second vector.

5. The model reasoning method according to any one of claims 1 to 3, characterized in that, Each block vector is determined based on the following steps: The key-value matrix is ​​divided into multiple block matrices along the sequence dimension. Average pooling is performed on the vectors within each segmentation matrix to obtain the corresponding segmented vectors.

6. The model reasoning method according to any one of claims 1 to 3, characterized in that, The step of inputting each second vector into the next layer and loading the key-value matrix of the target block vector corresponding to each second vector for model inference to obtain the third vector output by the next layer includes: Each second vector is input into the attention module of the next layer, and the key-value matrix of the target block vector corresponding to each second vector is loaded for attention calculation to obtain the attention calculation result corresponding to each second vector; The third vector is obtained based on the attention calculation results corresponding to each second vector.

7. The model reasoning method according to claim 5, characterized in that, The third vector is obtained based on the attention calculation results corresponding to each second vector, including: The attention calculation results corresponding to each second vector are added together or weighted and fused to obtain the third vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model reasoning method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the model reasoning method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the model reasoning method as described in any one of claims 1 to 7.

Images