Model reasoning method and device
By selecting the skipped target layer in the Transformer class big model and storing key-value information, the problem of speed and accuracy in long context inference is solved, and the inference effect that maintains high precision while improving speed is achieved.
Patent Information
- Application Number
- CN202411456272.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2024-12-24
AI Technical Summary
In the Transformer class large model, due to the need to store historical key-value information during long context inference, the key-value cache takes up a large space and slow calculation speed. The existing methods skip some layers to improve speed, but lead to a reduced inference accuracy.
By selecting the target layer to be skipped from the candidate target layer of the model, and determining and storing the key value information currently entered in the target layer, skipping attention calculations, performing forward propagation based on the results of the previous layer until all layers Reasoning results.
On the basis of improving the inference speed, the accuracy of the model is retained, the accuracy loss of the existing layer skipping method is avoided, and the accuracy of inference is ensured.
Smart Images
Figure CN119180342A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model reasoning technology, and in particular to a model reasoning method and device. Background Art
[0002] At present, in the structure of large Transformer models, the calculation mainly includes two parts: Attention and Feed Forward Neural Network (FFN). Moreover, large Transformer models are generally implemented in multiple layers, and each layer mainly includes Attention and FFN units. Usually, during reasoning, each input reasoning object (token) must go through Attention calculation and FFN calculation. In this way, when the context is relatively long, each layer of Attention needs to store the key value information (KV information) of the previous historical token, resulting in an increase in the storage space required for the key value cache (KVcache), a slower calculation speed of Attention, and a slower reasoning speed of the large model. In order to improve the reasoning speed, one solution is to selectively skip certain layers of Attention and FFN units for each token. The KVcache corresponding to the skipped layer lacks the KV information of this reasoning, resulting in low reasoning accuracy during the next reasoning. Summary of the invention
[0003] The present invention provides a model reasoning method and device, which are used to solve the problem that in the large model reasoning process of the prior art, it is impossible to improve the reasoning accuracy while improving the reasoning speed.
[0004] The present invention provides a model reasoning method, which includes the following steps.
[0005] Based on the current input inference object, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layers of the model.
[0006] When reasoning to the target layer, key value information of the current input reasoning object corresponding to the target layer is determined, and the key value information is stored in a key value cache corresponding to the target layer.
[0007] Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
[0008] According to a model inference method provided by the present invention, based on the current input inference object, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layer of the model, including: based on the current input inference object and the layer selection weight matrix obtained during the model training process, selecting the target layer that needs to be skipped during model inference corresponding to the current input inference object from the candidate target layer.
[0009] According to a model inference method provided by the present invention, the target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layer based on the current input inference object and the layer selection weight matrix obtained during the model training process, including the following steps.
[0010] Multiply the current input inference object and the layer selection weight matrix to obtain a 1×N weight vector, where N is the number of candidate target layers in the model, N is less than or equal to the total number of layers of the model, and each selection weight value in the weight vector corresponds to a candidate target layer.
[0011] Determine a candidate target layer corresponding to a selection weight value in the weight vector that is less than a preset weight threshold as the target layer.
[0012] According to a model inference method provided by the present invention, all candidate target layers in the model are divided into multiple groups, each group includes multiple candidate target layers, and all candidate target layers in each group share a key-value cache.
[0013] According to a model reasoning method provided by the present invention, based on the current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object is selected from the candidate target layers of the model, including the following steps.
[0014] Based on the current input inference object and the group selection weight matrix obtained in model training, a target group to be skipped is determined.
[0015] Based on the current input inference object and the layer selection weight matrix in the target group, the target layer to be skipped is determined from the target group. The layer selection weight matrix in the target group is obtained during model training and ensures that the number of target layers in the target group is less than the number of candidate target layers in the target group.
[0016] According to a model inference method provided by the present invention, when reasoning to the target layer, the key value information of the current input inference object corresponding to the target layer is determined, including: when reasoning to the first layer in each group, the reference key value information of the current input inference object corresponding to the first layer is determined, the reference key value information is stored in the key value cache within the group, and the reference key value information is determined as the key value information of the current input inference object corresponding to the target layer and the candidate target layer within the group.
[0017] According to a model reasoning method provided by the present invention, before selecting a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model based on the current input reasoning object, it also includes: determining that the first n layers in the model are non-candidate target layers, and the last N layers are the candidate target layers, wherein n is greater than or equal to 1, and n+N is the total number of layers of the model.
[0018] The present invention also provides a model reasoning device, comprising the following modules.
[0019] The target layer selection module is used to select, based on the current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model.
[0020] The key value information determination module is used to determine the key value information of the current input reasoning object corresponding to the target layer when reasoning to the target layer, and store the key value information in the key value cache corresponding to the target layer.
[0021] The forward reasoning module is used to skip the attention calculation of the target layer and perform the forward propagation calculation of the target layer based on the reasoning result of the previous layer corresponding to the target layer, until all layers of the model are passed to obtain the reasoning result of the current input reasoning object.
[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described model reasoning methods when executing the computer program.
[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the model reasoning methods described above.
[0024] The model inference method and device provided by the present invention selects a target layer that needs to be skipped during model inference corresponding to the current input inference object from the candidate target layer of the model based on the current input inference object; when inferring to the target layer, determines the key value information of the current input inference object corresponding to the target layer, and stores the key value information in the key value cache corresponding to the target layer; skips the attention calculation of the target layer, and based on the inference result of the previous layer corresponding to the target layer, performs the forward propagation calculation of the target layer until all layers of the model are passed to obtain the inference result of the current input inference object. Since when inferring to the target layer, for the current input inference object, the KV information calculation of the Attention unit in the target layer is not skipped, the accuracy of model inference is retained, and only the Attention calculation of the Attention unit is skipped, thereby improving the inference speed. On the basis of improving the inference speed, the inference accuracy is higher than that of the existing layer skipping method. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 It is one of the flowcharts of the model reasoning method in the related technology.
[0027] Figure 2 This is the second flowchart of the model reasoning method in the related technology.
[0028] Figure 3 It is a flow chart of the model reasoning method provided by the present invention.
[0029] Figure 4 It is a schematic diagram of determining the target layer to be skipped in the model inference method provided by the present invention.
[0030] Figure 5 It is a schematic diagram of multiple layers sharing a key-value cache in the model reasoning method provided by the present invention.
[0031] Figure 6 This is an example diagram of the reasoning process of the model reasoning method provided by the present invention, which has a four-layer structure.
[0032] Figure 7 It is a structural schematic diagram of the model reasoning device provided by the present invention.
[0033] Figure 8It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] In related technologies, large models are implemented in multiple layers, and each layer mainly includes Attention and FFN units. Among them, the Attention unit mainly includes three parameters when executing: query, key, and value. First, the input sequence (Input Embeddings) is converted into Query, Key, and Value vectors through a linear transformation (usually the multiplication of weight matrices WQ, WK, and WV), that is, KV information calculation, and then the model performs Attention calculation, that is, calculates the dot product of the Query vector and all Key vectors, and then calculates the weighted sum with the Value vector after softmax normalization to obtain the output sequence (Output Embeddings). In order to speed up the reasoning speed of the model, a solution to accelerate reasoning is as follows Figure 1 As shown, the Attention unit of the target layer in the model is skipped ( Figure 1 As shown in the dashed box in the figure, for each input inference object (i.e., input token) in the input sequence, the attention unit in the pre-selected target layer is skipped, that is, the attention unit in the target layer is not executed. Another solution to accelerate inference is as follows Figure 2 As shown in the figure, according to each token, through Route calculation, the target layer that the token needs to skip is first determined, and the Attention and FFN units of the target layer are skipped during reasoning. Figure 2 As shown on the left, for token X i , the current layer is its target layer, then the Attention unit (Self-attention and Normalize) of the current layer is skipped, and the forward propagation (Multi-layer Perceptron MLP) unit of the current layer is skipped. For token X i+1 , if the current layer is a non-target layer, normal reasoning is performed.
[0036] In the above two methods of accelerating reasoning, the execution process of the entire Attention unit of the target layer is skipped, which will cause the KV information of the one-step reasoning token to be missing in the KVcache corresponding to the target layer during the next reasoning, resulting in reduced reasoning accuracy. For example: Figure 2 In the case of token X i The current layer is skipped, and the KVcache corresponding to the current layer lacks X i KV information of token X i+1 , the inference accuracy decreases.
[0037] In view of the above technical problems existing in the prior art, the present invention provides a model reasoning method. Figure 3 It is a flow chart of the model reasoning method provided by the present invention, such as Figure 3 As shown, the method includes the following steps S310 to S330.
[0038] Step S310: Based on the current input inference object, select a target layer that needs to be skipped during model inference corresponding to the current input inference object from the candidate target layers of the model.
[0039] Specifically, for the current input reasoning object, the reasoning route calculation (Route calculation) is performed before reasoning to select the target layer corresponding to the current input reasoning object. Route calculation has the function of selecting suitable target layers for different input reasoning objects. Among them, all layers of the model can be candidate target layers, or only some layers can be candidate target layers.
[0040] Step S320: When reasoning to the target layer, determine the key value information of the current input reasoning object corresponding to the target layer, and store the key value information in the key value cache corresponding to the target layer.
[0041] Specifically, when reasoning to the target layer, the existing skipping method is to skip the Attention unit of the entire target layer. In this embodiment, the key value information of the current input reasoning object corresponding to the target layer is determined, that is, the KV information of the current input reasoning object is calculated, and the KV information is stored in the key value cache corresponding to the target layer. Since the KV information calculation of the Attention unit in the target layer is not skipped, the KV information of the current input reasoning object corresponding to the target layer is retained, that is, the complete KV information of the historical reasoning object is retained in the key value cache corresponding to each layer of the model, so that the reasoning accuracy of each reasoning is retained, and the final reasoning result has a higher accuracy.
[0042] Step S330: Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
[0043] Specifically, after calculating and storing the KV information of the current input reasoning object corresponding to the target layer, the Attention calculation of the Attention unit in the target layer is skipped, and the forward propagation calculation of the target layer is directly performed based on the reasoning result of the previous layer corresponding to the target layer, until all layers of the model are passed to obtain the reasoning result of the current input reasoning object. Since the Attention calculation of the Attention unit is skipped, the amount of calculation in the reasoning process is reduced, thereby improving the reasoning speed.
[0044] In the model inference method of this embodiment, when reasoning to the target layer, for the current input reasoning object, the KV information calculation of the Attention unit in the target layer is not skipped, the accuracy of the model reasoning is retained, and only the Attention calculation of the Attention unit is skipped, thereby improving the reasoning speed. On the basis of improving the reasoning speed, the reasoning accuracy is higher than that of the existing layer skipping method. Moreover, experiments show that during model reasoning, some Attention calculations can be skipped without causing loss of accuracy, but after skipping the FFN unit, it will cause a significant decrease in accuracy in some reasoning tasks. In this embodiment, the FFN unit of the target layer is not skipped, thereby ensuring a high reasoning accuracy of the model.
[0045] It should be noted that the model reasoning method of this embodiment is not only applicable to the reasoning of large models, but also to the reasoning of special multi-layer models.
[0046] In some embodiments, the step S310 specifically includes: based on the current input inference object and the layer selection weight matrix obtained during the model training process, selecting a target layer that needs to be skipped during model inference corresponding to the current input inference object from the candidate target layer.
[0047] Among them, the layer selection weight matrix router weight is obtained during the model training process, which is an N×N matrix, where N is the number of candidate target layers in the model. The layer selection weight matrix is updated as a model parameter during the model training process until the model training is completed. It can be understood that: in one iteration, the corresponding target layer is skipped after calculation according to the sample inference object and the router weight, and the model obtains the prediction result, and after calculating through the loss function, the layer selection weight matrix is updated. In the next iteration, the sample inference object skips different target layers after calculation through the updated layer selection weight matrix, and the model obtains the prediction result, and calculates through the loss function until the loss function converges, and the final layer selection weight matrix is determined.
[0048] In this embodiment, the Route calculation is performed through the current input token and the layer selection weight matrix router weight to obtain the target layer that the current input reasoning object needs to skip. Since the layer selection weight matrix is obtained during the model training process, it has a high output accuracy after the model training is completed. In the model reasoning stage, the current input reasoning object and the layer selection weight matrix are used to calculate the target layer to be skipped, so that the reasoning accuracy can maintain a high accuracy during model training.
[0049] In some embodiments, based on the current input inference object and the layer selection weight matrix obtained during the model training process, selecting a target layer that needs to be skipped during model inference corresponding to the current input inference object from the candidate target layer includes the following steps.
[0050] Multiply the current input inference object and the layer selection weight matrix to obtain a 1×N weight vector, where N is the number of candidate target layers in the model, N is less than or equal to the total number of layers of the model, and each selection weight value in the weight vector corresponds to a candidate target layer.
[0051] Determine a candidate target layer corresponding to a selection weight value in the weight vector that is less than a preset weight threshold as the target layer.
[0052] Specifically, Figure 4As shown, during Route calculation, the current input reasoning object token is first converted into a 1×N hidden vector hidden. The hidden vector hidden is multiplied by the N×N layer selection weight matrix router weight to obtain a 1×N weight vector. Each element in the weight vector corresponds to a candidate target layer. The value of the element is the selection weight value, indicating whether the corresponding candidate target layer is determined as the target layer. The preset weight threshold can be 0.4~0.6, preferably 0.5, that is, the candidate target layer with a weight value less than 0.5 is selected as the target layer. It can be understood that: each column of the layer selection weight matrix routerweight corresponds to a candidate target layer, for example: the first column corresponds to the first layer, and the hidden vector hidden is multiplied by the first column of the layer selection weight matrix router weight to obtain the selection weight value of 0.14 for the first layer, indicating that the first layer is the target layer that needs to be skipped.
[0053] Figure 4 In the example of N=4, the selection weight values of the 0th to 3rd elements in the weight vector correspond to the first to fourth candidate target layers respectively. The selection weight values corresponding to the first and fourth layers are less than 0.5 and are determined as target layers. During model inference, for the current input token, the Attention calculation of the Attention units of the first and fourth layers needs to be skipped.
[0054] Among them, the preset weight threshold is a fixed value during the model training and reasoning process, and a relatively compromise value is selected, for example: 0.5, so that the model converges better during training, and avoids subjectively setting too large or too small values to affect the training and reasoning accuracy of the model.
[0055] In some embodiments, all candidate target layers in the model are divided into multiple groups, each group includes multiple candidate target layers, and all candidate target layers in each group share a key-value cache, for example: Figure 5 As shown in the figure, two or three candidate target layers can be grouped together. Since the candidate target layers in each group share a key-value cache in the group, it is not necessary for each layer to cache the KV information of the historical token, thereby saving the storage space required for the model during inference.
[0056] In some embodiments, based on grouping the candidate target layers and the candidate target layers in the group sharing a key-value cache, step S310 specifically includes the following steps.
[0057] Step 1: Determine the target group to be skipped based on the current input inference object and the group selection weight matrix obtained in model training.
[0058] Specifically, the group selection weight matrix is determined during the model training process. For example, if there are M groups in total, then the group selection weight matrix is an N×M matrix. During route calculation, the current input inference object token is first converted into a 1×N latent vector. The 1×N latent vector is multiplied by the N×M group selection weight matrix to obtain a 1×M group weight vector. The group corresponding to the group selection weight value less than the group selection threshold (for example, 0.5) in the group weight vector is determined as the target group. The entire target group determination process is the same as Figure 4 similar.
[0059] like Figure 6 As shown, taking the four-layer model as an example, all layers are candidate target layers, the first candidate target layer layer1 and the second candidate target layer layer2 are a group, sharing the key cache KVcache1, the third candidate target layer layer3 and the fourth candidate target layer layer4 are a group, sharing the key cache KVcache2. After step 1, it is determined that both groups are target groups, that is, the Attention calculation of the Attention units in some layers needs to be skipped in both groups.
[0060] Step 2: Based on the current input reasoning object and the layer selection weight matrix in the target group, determine the target layer to be skipped from the target group. The layer selection weight matrix in the target group is obtained during model training, and the number of target layers in the target group is ensured to be less than the number of candidate target layers in the target group, so as to ensure that at least one layer of the Attention unit in the target group needs to perform Attention calculation, thereby ensuring the reasoning accuracy of the model. Because, considering the model reasoning effect, the Attention calculation of each group needs to play the role of extracting position information. If all Attention calculations in each group are skipped, even if the key value information KV is retained, it will cause a decrease in accuracy.
[0061] Specifically, for the non-target group (in which all candidate target layers are inferred normally), it is not necessary to determine the target layer. Only the target layer in the target group is determined, that is, the Route calculation is performed. If there are K candidate target layers in the target group, during the Route calculation, the current input inference object token is first converted into a 1×N latent vector. The 1×N latent vector is multiplied by the layer selection weight matrix in the N×K target group to obtain a 1×K layer weight vector. The candidate target layer corresponding to the layer selection weight value in the layer weight vector that is less than the layer selection threshold (for example: 0.5) is determined as the target layer.
[0062] In this embodiment, since the candidate target layers are grouped and the key value cache is shared within the group, when selecting the target layer, it is necessary to first select the target group and then select the target layer from the target group. This avoids the problem that during the model training process, the direct selection of the target layer will result in all the candidate target layers in a group being determined as the target layers, that is, all the layers in a group will skip the Attention calculation of the Attention unit, thereby affecting the model reasoning accuracy.
[0063] In some embodiments, in step S320, when reasoning to the target layer, determining the key value information of the current input reasoning object corresponding to the target layer specifically includes: when reasoning to the first layer in each group, determining the reference key value information of the current input reasoning object corresponding to the first layer, storing the reference key value information in the key value cache within the group, and determining the reference key value information as the key value information of the current input reasoning object corresponding to the target layer and the candidate target layer within the group.
[0064] like Figure 5 and 6 As shown, for all layers of each group, when reasoning to the first layer, the Attention unit of the first layer calculates KV information for the current input reasoning object, and stores the calculated reference KV information in the key value cache within the group. The subsequent non-target layers in the group use the KV information in the key value cache within the group to perform Attention calculations, and the subsequent target layers in the group can also skip the KV information calculation and Attention calculation of the Attention unit. In this embodiment, since the key value cache is shared within the group, it is not necessary for each layer in the group to perform KV information calculations, which further improves the model reasoning speed while ensuring the model reasoning accuracy.
[0065] A large number of experimental studies on model training and reasoning have shown that the KV information of the initial layer or the first few layers of the model is more important for reasoning accuracy. Therefore, in some embodiments, before step S310, the model reasoning method also includes: determining that the first n layers in the model are non-candidate target layers, and the last N layers are the candidate target layers, wherein n is greater than or equal to 1, and n+N is the total number of layers of the model. That is, normal reasoning is performed for the first n layers, and for the last N layers, reasoning is performed by skipping the Attention calculation of the Attention unit in the target layer, or skipping the Attention calculation of the Attention unit in the target layer in combination with a shared key-value cache, thereby further improving the reasoning accuracy of the model.
[0066] For example, if the model has 32 layers, the first 4 layers can be inferred normally, each layer has its own KVcache, and the next 28 layers are inferred by skipping the target layer and sharing the KVcache. Preferably, n is less than N, which can more effectively improve the inference speed of the model while ensuring the inference accuracy.
[0067] The model reasoning device provided by the present invention is described below. The model reasoning device described below and the model reasoning method described above can be referenced to each other.
[0068] The model inference device of this embodiment is as follows Figure 7 As shown, it includes the following functional modules.
[0069] The target layer selection module 710 is used to select, based on the current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model.
[0070] The key value information determination module 720 is used to determine the key value information of the current input reasoning object corresponding to the target layer when reasoning to the target layer, and store the key value information in the key value cache corresponding to the target layer.
[0071] The forward reasoning module 730 is used to skip the attention calculation of the target layer and perform the forward propagation calculation of the target layer based on the reasoning result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the reasoning result of the current input reasoning object.
[0072] The model inference device of this embodiment, when inferring to the target layer, does not skip the KV information calculation of the Attention unit in the target layer for the current input inference object, retains the accuracy of model inference, and only skips the Attention calculation of the Attention unit, thereby improving the inference speed. On the basis of improving the inference speed, the inference accuracy is higher than that of the existing layer skipping method. Moreover, experiments show that during model inference, some Attention calculations can be skipped without causing loss of accuracy, but skipping the FFN unit will cause a significant decrease in accuracy in some tasks. In this embodiment, the FFN unit of the target layer is not skipped, thereby ensuring a high inference accuracy of the model.
[0073] In some embodiments, the target layer selection module 710 is specifically used to select a target layer that needs to be skipped during model reasoning corresponding to the current input inference object from the candidate target layers based on the current input inference object and the layer selection weight matrix obtained during the model training process.
[0074] In some embodiments, the target layer selection module 710 is specifically used to multiply the current input inference object and the layer selection weight matrix to obtain a 1×N weight vector, wherein N is the number of candidate target layers in the model, N is less than or equal to the total number of layers of the model, and each selection weight value in the weight vector corresponds to a candidate target layer; and the candidate target layer corresponding to the selection weight value in the weight vector that is less than a preset weight threshold is determined as the target layer.
[0075] In some embodiments, all candidate target layers in the model are divided into multiple groups, each group includes multiple candidate target layers, and all candidate target layers in each group share a key-value cache.
[0076] In some embodiments, the target layer selection module 710 is specifically used to determine the target group to be skipped based on the current input inference object and the group selection weight matrix obtained in the model training; based on the current input inference object and the layer selection weight matrix in the target group, determine the target layer to be skipped from the target group, the layer selection weight matrix in the target group is obtained during model training, and ensures that the number of target layers in the target group is less than the number of candidate target layers in the target group.
[0077] In some embodiments, the key value information determination module 720 is specifically used to determine the reference key value information of the current input reasoning object corresponding to the first layer in each group when reasoning to the first layer, store the reference key value information in the key value cache within the group, and determine the reference key value information as the key value information of the current input reasoning object corresponding to the target layer and the candidate target layer within the group.
[0078] In some embodiments, the model reasoning device also includes: a candidate target layer selection module, which is used to determine that the first n layers in the model are non-candidate target layers and the last N layers are the candidate target layers before selecting the target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model based on the current input reasoning object, where n is greater than or equal to 1, and n+N is the total number of layers of the model.
[0079] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the model reasoning method, which includes: Based on the current input inference object, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layers of the model.
[0080] When reasoning to the target layer, key value information of the current input reasoning object corresponding to the target layer is determined, and the key value information is stored in a key value cache corresponding to the target layer.
[0081] Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
[0082] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0083] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the model reasoning method provided by the above methods, the method comprising: Based on the current input inference object, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layers of the model.
[0084] When reasoning to the target layer, key value information of the current input reasoning object corresponding to the target layer is determined, and the key value information is stored in a key value cache corresponding to the target layer.
[0085] Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
[0086] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model reasoning method provided by the above methods, the method comprising: Based on the current input inference object, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layers of the model.
[0087] When reasoning to the target layer, key value information of the current input reasoning object corresponding to the target layer is determined, and the key value information is stored in a key value cache corresponding to the target layer.
[0088] Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model reasoning method, characterized in that: include: Based on the current input reasoning object, select a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model; When reasoning to the target layer, determining the key value information of the current input reasoning object corresponding to the target layer, and storing the key value information in the key value cache corresponding to the target layer; Skip the attention calculation of the target layer, and perform the forward propagation calculation of the target layer based on the inference result of the previous layer corresponding to the target layer, until passing through all layers of the model to obtain the inference result of the current input inference object.
2. The model reasoning method according to claim 1, characterized in that: The selecting, based on the current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model includes: Based on the current input inference object and the layer selection weight matrix obtained in the model training process, a target layer that needs to be skipped during model inference corresponding to the current input inference object is selected from the candidate target layers.
3. The model reasoning method according to claim 2, characterized in that: The selecting, based on the current input inference object and the layer selection weight matrix obtained in the model training process, a target layer to be skipped during model inference corresponding to the current input inference object in the candidate target layer comprises: Multiply the current input inference object and the layer selection weight matrix to obtain a 1×N weight vector, where N is the number of candidate target layers in the model, N is less than or equal to the total number of layers in the model, and each selection weight value in the weight vector corresponds to a candidate target layer; Determine a candidate target layer corresponding to a selection weight value in the weight vector that is less than a preset weight threshold as the target layer.
4. The model reasoning method according to claim 1, characterized in that: All candidate target layers in the model are divided into multiple groups, each group includes multiple candidate target layers, and all candidate target layers in each group share a key-value cache.
5. The model reasoning method according to claim 4, characterized in that: The selecting, based on the current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model includes: Determine a target group to be skipped based on the current input inference object and a group selection weight matrix obtained in model training; Based on the current input inference object and the layer selection weight matrix in the target group, the target layer to be skipped is determined from the target group. The layer selection weight matrix in the target group is obtained during model training and ensures that the number of target layers in the target group is less than the number of candidate target layers in the target group.
6. The model reasoning method according to claim 4, characterized in that: When reasoning to the target layer, determining the key value information of the current input reasoning object corresponding to the target layer includes: When reasoning to the first layer in each group, determine the reference key value information of the current input reasoning object corresponding to the first layer, store the reference key value information in the key value cache within the group, and determine the reference key value information as the key value information of the current input reasoning object corresponding to the target layer and the candidate target layer within the group.
7. The model reasoning method according to any one of claims 1 to 6, characterized in that: Before selecting, based on the current input inference object, from the candidate target layers of the model, a target layer that needs to be skipped during model inference corresponding to the current input inference object, the method further includes: It is determined that the first n layers in the model are non-candidate target layers, and the last N layers are the candidate target layers, wherein n is greater than or equal to 1, and n+N is the total number of layers of the model.
8. A model reasoning device, characterized in that: include: A target layer selection module is used to select, based on a current input reasoning object, a target layer that needs to be skipped during model reasoning corresponding to the current input reasoning object from the candidate target layers of the model; A key value information determination module, used for determining the key value information of the current input reasoning object corresponding to the target layer when reasoning to the target layer, and storing the key value information in the key value cache corresponding to the target layer; The forward reasoning module is used to skip the attention calculation of the target layer and perform the forward propagation calculation of the target layer based on the reasoning result of the previous layer corresponding to the target layer, until all layers of the model are passed to obtain the reasoning result of the current input reasoning object.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model reasoning method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning method according to any one of claims 1 to 7 is implemented.