Fault-tolerant methods, products, devices, and media for key-value caching in models

Abstract

The present application discloses a fault-tolerant method, product, device and medium for a key-value cache in a model, which relates to the field of computer technology. The method comprises reading the key vectors of multiple words in the key-value cache of each layer in multiple attention layers layer by layer and performing detection. When it is detected that there is a reading error in the key vector of the first word corresponding to the current layer, the attention score of the current layer is corrected according to the attention importance of the first word in the previous layer. The method solves the technical problem of poor stability and reliability of model reasoning caused by key vector reading errors in the key-value cache, and achieves the technical effect of improving the fault tolerance capability of the key-value cache, thereby improving the stability and reliability of model reasoning.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault-tolerant method, product, device, and medium for key-value caching in a model. Background Art

[0002] Pre-trained language models based on the Transformer architecture have achieved remarkable success in fields such as natural language processing and content generation. Key-value caching (KVCache) technology has been widely adopted to improve model inference efficiency. KVCache stores the intermediate states of the attention mechanism, namely the key and value vectors, avoiding repeated calculations of historical sequences when generating each new token. However, as the length of the input sequence increases and the batch size increases, the memory resources occupied by KVCache expands dramatically, potentially exceeding the memory required for the model parameters themselves. This large-scale memory usage makes KVCache particularly sensitive to memory hardware errors, especially soft errors such as bit flips.

[0003] Bit-flip errors can cause abnormalities in floating-point values ​​(such as FP16 and BF16) stored in KVCache, such as sign bit flips or exponent bit flips, resulting in key vectors with abnormal amplitudes. Reading abnormal key vectors can severely interfere with the calculation of the attention mechanism, significantly degrading model output quality and affecting the stability and reliability of model inference. Summary of the Invention

[0004] The present application provides a fault-tolerant method, product, device, and medium for a key-value cache in a pre-trained language model, to at least address the problem of poor stability and reliability of model inference caused by key vector read errors in the key-value cache.

[0005] This application provides a fault-tolerant method for key-value caching in a model, including:

[0006] Read the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer of the multi-layer attention layer of the pre-trained language model layer by layer;

[0007] In the case of reading the key-value vectors of the multiple word-units corresponding to the current layer, detecting the key vectors of the multiple word-units corresponding to the current layer to detect whether the key vectors have a reading error caused by a storage medium failure;

[0008] When it is detected that there are no reading errors in the key vectors of multiple word units corresponding to the current layer, the attention score of the current layer is calculated according to the key vectors of the multiple word units corresponding to the current layer and the context input vector;

[0009] When a read error is detected in a key vector of a first word-gram among multiple words-grams corresponding to a current layer, determining the attention importance of the first word-gram in a previous layer, and correcting an attention score of the current layer according to the attention importance of the first word-gram in the previous layer;

[0010] The context output vector of the current layer is determined based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is the context input vector of the next layer, and the context output vector of the last layer is used to generate the inference result corresponding to the input text.

[0011] The present application also provides a computer program product, comprising:

[0012] The reading module is used to read the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer in the multi-layer attention layer of the pre-trained language model layer by layer;

[0013] a detection module, configured to detect, when reading the key-value vectors of the multiple word-units corresponding to the current layer, the key vectors of the multiple word-units corresponding to the current layer, to detect whether the key vectors have a reading error caused by a storage medium failure;

[0014] A calculation module, configured to calculate an attention score of the current layer based on the key vectors of the multiple word units corresponding to the current layer and the context input vector when detecting that there are no reading errors in the key vectors of the multiple word units corresponding to the current layer;

[0015] a correction module, configured to, upon detecting a read error in a key vector of a first word-gram among a plurality of words-grams corresponding to a current layer, determine the attention importance of the first word-gram in a previous layer, and correct an attention score of the current layer based on the attention importance of the first word-gram in the previous layer;

[0016] A determination module is used to determine the context output vector of the current layer based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is the context input vector of the next layer, and the context output vector of the last layer is used to generate the inference result corresponding to the input text.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the fault-tolerant method of the key-value cache in any of the above models when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the fault-tolerant method of the key-value cache in any of the above models are implemented.

[0019] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the fault-tolerant method of the key-value cache in any of the above models.

[0020] Through this application, during the reasoning process of the model, the key-value vectors of multiple words in the key-value cache of each layer in multiple attention layers are read layer by layer and detected. When it is detected that there is a reading error in the key vector of the first word corresponding to the current layer, the attention score of the current layer is corrected according to the attention importance of the first word in the previous layer. Therefore, the problem of poor stability and reliability of model reasoning caused by the reading error of the key vector in the key-value cache can be solved, and the technical effect of timely correcting the attention score when there is a reading error in the key vector in the key-value cache is achieved, improving the fault tolerance of the key-value cache, and thus improving the stability and reliability of model reasoning is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a fault-tolerant method for key-value caching in a model provided in an embodiment of the present application;

[0023] Figure 2 Another flowchart of a fault-tolerant method for key-value caching in a model provided in an embodiment of the present application;

[0024] Figure 3 A block diagram of a computer program product provided in an embodiment of the present application.

[0025] Reference numerals:

[0026] Among them, there are a computer program product 10; a reading module 100; a detection module 200; a calculation module 300; a correction module 400; and a determination module 500. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] An embodiment of the present application provides a fault-tolerant method for key-value caching in a model, and describes the method in detail in conjunction with the execution flow of the fault-tolerant method for key-value caching in a model.

[0031] Specifically, Figure 1 The present invention provides a flowchart of a fault-tolerant method for key-value caching in a model according to an embodiment of the present application.

[0032] like Figure 1 As shown, the fault-tolerant method of key-value cache in the model includes steps 110 to 150.

[0033] Step 110: Read the key-value vectors of multiple word units related to the input text in the key-value cache of each layer in the multi-layer attention layer of the pre-trained language model layer by layer.

[0034] The pre-trained language model can be a large language model. A large language model, also known as a large language model, is a deep learning model trained on massive amounts of text data and possessing billions or even hundreds of billions of parameters. Its core goal is to achieve natural language generation, comprehension, and interaction by understanding the patterns and regularities of human language.

[0035] The basic architecture of the pre-trained language model is Transformer. The Transformer architecture gradually extracts and deepens the semantic representation of the text through multi-layer stacking. The pre-trained language model can include an embedding layer, multi-layer attention layers, and an output layer. During the inference process of the pre-trained language model, the input text is input to the embedding layer, which converts the input text into a vector representation of a word sequence. The vector representation of the word sequence serves as the context input vector of the first attention layer. In the multi-layer attention layer, each attention layer calculates the relationship between each word in the word sequence and all other words based on its context input vector, and generates a context output vector. The context output vector serves as the context input vector of the next attention layer. The output layer converts the context output vector of the last attention layer into the output of the model, and the output of the model is the inference result corresponding to the input text. In addition, the context output vector of the last attention layer can also continue to serve as the context input vector of the first attention layer to continue generating the inference result corresponding to the input text.

[0036] Each layer in the multi-layer attention layer corresponds to an independent key-value cache, which stores the key-value vector of the generated token. The key-value vector includes a key vector and a value vector.

[0037] In each attention layer, for generated tokens, the query (Q) vector is calculated based on the context input vector of the attention layer, and the key (K) vector and value (V) vector of the token are read from the key-value cache of the attention layer. For ungenerated tokens (new tokens), the query vector, key vector, and value vector are calculated based on the context input vector of the attention layer, and the key vector and value vector of the token are stored in the key-value cache of the attention layer. The query vector represents the context information that currently needs to be paid attention to, the key vector represents the identification information of each token in the token sequence, and the value vector represents the actual information of each token in the token sequence.

[0038] In this embodiment, during the inference process of the pre-trained language model, the vector of the word-meta sequence (including multiple word-meta) converted from the input text through the embedding layer is used as the context input vector for the first attention layer. After receiving its context input vector, the first attention layer reads the key-value vectors of multiple word-meta related to the input text from its key-value cache, performs detection and other processing, and generates a context output vector. The context output vector of the first attention layer serves as the context input vector for the second attention layer. After receiving its context input vector, the second attention layer reads the key-value vectors of multiple word-meta related to the input text from its key-value cache, performs detection and other processing, and generates a context output vector. The context output vector of the second attention layer serves as the context input vector for the third attention layer. This continues in this manner until the last attention layer receives its context input vector, reads the key-value vectors of multiple word-meta related to the input text from its key-value cache, performs detection and other processing, and generates a context output vector. The context output vector of the last attention layer is converted into the inference result corresponding to the input text through the output layer. The context output vector of the last attention layer can also be used as the context input vector of the first attention layer to continue generating the inference result corresponding to the input text until the inference of the pre-trained language model is completed.

[0039] Step 120 : When reading the key-value vectors of multiple word-units corresponding to the current layer, detect the key vectors of the multiple word-units corresponding to the current layer to check whether there are any reading errors in the key vectors caused by a storage medium failure.

[0040] The storage medium of the key-value cache may fail when affected by external factors such as cosmic rays and radiation, or when there are defects in the hardware. These failures may occur in the form of transient failures or silent failures. A transient failure is a temporary data error caused by external factors that does not cause permanent damage to the hardware. For example, a correct key-value vector is written to the key-value cache, but an error occurs when the key-value vector is read from the key-value cache. However, this error is transient and can be corrected by re-reading. A silent failure is an error that is not immediately detected when it occurs, resulting in data corruption but the system continues to run, which may cause subsequent logical errors. For example, a key-value vector is incorrectly written to the key-value cache, or hardware aging causes corruption in the stored data, resulting in an incorrect key-value vector being read from the key-value cache.

[0041] Incorrect key-value vectors can affect the calculation of the attention mechanism, thereby affecting the model output. Therefore, when reading the key-value vectors of multiple word units in the key-value cache of multiple attention layers layer by layer, each time the key-value vectors of multiple word units in the key-value cache of an attention layer are read, the key-value vectors of the multiple word units in the key-value cache of that attention layer are tested, that is, whether the read key-value vectors contain read errors caused by storage medium failure. This test includes key vector detection. Among them, the key vector detection can be performed before calculating the attention score of the attention layer.

[0042] Each element of the key vector is a floating-point number. A floating-point number consists of a sign, an exponent, and a mantissa. The sign indicates the sign of the value (0 for positive, 1 for negative), the exponent indicates the range of the value (i.e., the exponent in scientific notation), and the mantissa indicates the significant digits of the value (i.e., the decimal part in scientific notation).

[0043] A storage medium failure of the key-value cache can easily lead to a read error in the key vector read from the key-value cache, such as a bit reversal error, which causes the sign bit inversion or the exponent bit key bit reversal in the floating-point number of the key vector, that is, the key vector read from the key-value cache is wrong. The wrong key vector will seriously interfere with the calculation of the attention mechanism and affect the model output results. Therefore, after reading the key-value vectors of multiple word units corresponding to the current layer, this embodiment detects the key vectors of the multiple word units read in the current layer. If there is no read error in the key vectors of the multiple word units read in the current layer, the calculation of the attention score of the current layer is performed normally; if there is a read error in the key vector of at least one word unit read in the current layer, the attention score of the current layer is corrected to avoid the abnormal key vector affecting the calculation of the attention mechanism, thereby avoiding affecting the model output results.

[0044] In some embodiments, detecting the key vectors of the multiple word-grams corresponding to the current layer in step 120 includes:

[0045] Compare the key vectors of multiple word units corresponding to the current layer with the key vector threshold respectively;

[0046] If the key vectors of the multiple word units corresponding to the current layer do not exceed the key vector threshold, it is determined that the key vectors of the multiple word units corresponding to the current layer do not have a reading error;

[0047] If the key vector of the first word element corresponding to the current layer exceeds the key vector threshold, it is determined that there is a read error in the key vector of the first word element corresponding to the current layer.

[0048] Among them, the key vector threshold Threshold K Refers to the normal operation threshold of each element of the key vector. Key vector threshold Threshold KIt can be set before actual inference is performed on the pre-trained language model. When detecting the key vectors of multiple words, the current layer Each element (or key bit, such as the index part) of the key vector of each corresponding word element and the key value vector threshold Threshold K Compare. If the current layer Each element of the key vector of each corresponding word does not exceed the key vector threshold Threshold K , it indicates that the current layer The key vectors of the corresponding word units are running normally and there is no reading error; if the current layer At least one element of the key vector of at least one corresponding word-gram (such as the first word-gram) exceeds the key vector threshold Threshold K , it indicates that the current layer There is a read error in the key vector of the corresponding first word.

[0049] Step 130: When it is detected that there are no reading errors in the key vectors of multiple word units corresponding to the current layer, the attention score of the current layer is calculated based on the key vectors of the multiple word units corresponding to the current layer and the context input vector.

[0050] When the current layer is detected If there are no reading errors in the key vectors of the corresponding multiple words, the current layer is executed normally. The calculation of the attention score of the current layer The attention score is corrected.

[0051] For example, according to the current layer The context input vector can be used to calculate the query vector Q l . According to the query vector Q l and from the current layer The key vector read from the key-value cache K l , calculate the current layer Attention score AttentionScores l ,Right now .in, is the dimension of the representation vector of the intermediate layer, and no special fault tolerance processing is required. Attention score AttentionScores l Include current layer The attention score of each corresponding word.

[0052] Step 140: When a reading error is detected in the key vector of the first word among the multiple words corresponding to the current layer, the attention importance of the first word in the previous layer is determined, and the attention score of the current layer is corrected based on the attention importance of the first word in the previous layer.

[0053] When the current layer is detected The corresponding first word token The key vector K (l,i) In case of a reading error, the first word token On the previous level The importance of attention, , for the current layer The attention score is corrected to avoid abnormal key vectors K (l,i) Affects the current layer Attention score.

[0054] In some embodiments, determining the attention importance of the first word in the previous layer includes:

[0055] Sort the attention scores of multiple words corresponding to the previous layer in descending order;

[0056] If the attention score of the first word corresponding to the previous layer is ranked in the first target digit, it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold;

[0057] If the attention score of the first word corresponding to the previous layer is not ranked in the top target digits, it is determined that the attention importance of the first word in the previous layer is less than the first importance threshold.

[0058] Since the current layer The attention scores of the previous layer have not been calculated, and the attention distribution between layers is highly similar, so The attention score of On the previous level The importance of attention to indirectly determine the first word token In the current layer The importance of attention.

[0059] Previous level Attention score AttentionScores l-1 Including the previous layer The attention score of each corresponding word. The corresponding first word token The higher the attention score of On the previous level The higher the importance of attention.

[0060] Set the first importance threshold top k , indicating that the target number of digits is , It can be the upper layer The corresponding number of words is 20%. The larger the better, but due to the sparsity of attention, the marginal benefit increases with decreases with the increase of .

[0061] Detect the first word token The attention score of the previous layer The sorting of attention scores, if the first word token The attention score is ranked first Then determine the first word token On the previous level The attention importance of is greater than or equal to the first importance threshold top k , that is, the first word token On the previous level The attention importance is high, indirectly determining the first word token In the current layer The attention importance of The attention score is not ranked in the top Then determine the first word token On the previous level The attention importance of is less than the first importance threshold top k , that is, the first word token On the previous level The attention importance of is low, and the first word token is indirectly determined In the current layer The importance of attention is low.

[0062] In this embodiment, the first word token On the previous level The attention score of On the previous level The importance of attention is used to determine the first word token more directly and efficiently by using the inter-layer attention similarity. In the current layer The importance of attention.

[0063] In some embodiments, modifying the attention score of the current layer according to the attention importance of the first word in the previous layer includes:

[0064] When it is determined that the attention importance of the first word in the previous layer is less than the first importance threshold, the key vector of the first word corresponding to the current layer is discarded, and the attention score of the current layer is calculated based on the key vectors of other words corresponding to the current layer and the context input vector;

[0065] When it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold, the attention score of the previous layer is used as the attention score of the current layer.

[0066] If the first word token On the previous level The attention importance of is less than the first importance threshold, which means that the first word token On the previous level The attention importance of the first word token is low. In the current layer The attention importance of may also be low, or even if the attention importance is high, but its key vector is damaged, the first word token is directly discarded The key vector of , or the first word token The attention contribution of is set to negative infinity or a very small negative value, so that its weight after subsequent Softmax normalization is 0. The key vector does not participate in the current layer The calculation of the attention score of the current layer The key vectors of the corresponding other words do not have any reading errors and participate in the current layer normally. The calculation of the attention score of the current layer The key vectors and query vectors of the corresponding other words are calculated at the current layer The attention score of the current layer Correction of attention score.

[0067] This embodiment directly ignores the first word token Key vectors are used to avoid the introduction of noise by forced use, thereby avoiding the interference of erroneous key vectors and avoiding complex recovery operations or full backups, significantly reducing the computing and I / O overhead introduced by fault tolerance.

[0068] If the first word token On the previous level The attention importance of is greater than or equal to the first importance threshold, which means that the first word token On the previous level The attention importance of the first word token is high. In the current layer The attention importance of may also be high. The corresponding first word token The key vector of the current layer is corrupted and cannot be used for Perform reliable attention score calculations, so the current layer is no longer used The key vectors of the corresponding multiple words (including the key vectors without reading errors) and the query vector are used to calculate the complete and independent attention scores. The corresponding first word token The attention of is very important and cannot be discarded directly, but the attention distribution between layers is similar, so it is directly used or adapted to the previous layer. Attention score AttentionScores l-1 , the previous layer Attention score AttentionScores l-1 As current layer Attention score AttentionScores l = AttentionScores l-1 , so as to realize the current layer Compared to calculating the current layer based on incomplete or wrong key vectors The attention score of the previous layer is used directly The attention score is more reliable.

[0069] Related technologies primarily utilize inter-layer similarity in model design (such as parameter sharing) or indirect transfer of sparsity patterns. However, this embodiment directly uses the previous layer's attention score as the current layer's attention score when a key vector read error occurs. This is a more radical but more effective application of inter-layer similarity, especially when the current layer's key information is corrupted. This prevents the attention mechanism from completely failing due to this corruption.

[0070] This embodiment avoids using abnormal key vectors to calculate the attention score of the current layer, reducing error propagation. Moreover, since the calculation dependence on abnormal key vectors is reduced, it can also bring certain computing and I / O optimizations.

[0071] In some embodiments, when a read error is detected in the key vector of the first word corresponding to the current layer, the method further includes:

[0072] If the attention importance of the first word in the previous layer is not determined, the key vector of the first word corresponding to the current layer is discarded, and the attention score of the current layer is calculated based on the key vectors of other words corresponding to the current layer and the context input vector.

[0073] In the current layer The corresponding first word token In the event of a read error, by obtaining the previous layer The attention score of On the previous level The importance of attention, if the previous layer cannot be obtained The attention score of is the first layer, i.e. , or the previous level In the event of a large-scale key vector reading error, the previous layer cannot be obtained. The attention score of On the previous level The importance of attention.

[0074] In this case, the first word token is discarded directly. The value vector of , or the first word token The attention contribution of is set to negative infinity or a very small negative value, so that its weight after subsequent Softmax normalization is 0. The value vector does not participate in the current layer The calculation of the attention score of the current layer The key vectors of the corresponding other words do not have any reading errors and participate in the current layer normally. The calculation of the attention score of the current layer Correction of attention score.

[0075] In this embodiment, the current layer is calculated When the attention score is , the first word token is ignored directly This is a conservative correction strategy that sacrifices potentially important information but avoids forcibly introducing noise, thereby avoiding the interference of erroneous key vectors and saving some computational overhead.

[0076] Step 150: Determine the context output vector of the current layer based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is the context input vector of the next layer, and the context output vector of the last layer is used to generate the inference result corresponding to the input text.

[0077] The current layer to be calculated or modified The attention score of the current layer includes The attention score of each word. The current layer calculated or modified The attention score of the current layer is normalized by Softmax to obtain The corresponding attention weights of each word unit, for the current layer The corresponding attention weights of each word and the read value vector are weighted summed to calculate the current layer The context output vector of the current layer The context output vector of the next layer The context input vector of .

[0078] In this embodiment, the current layer If there is a reading error in the key vector of the corresponding first word, the current layer The attention score is corrected to prevent abnormal key vectors from interfering with the calculation of the attention mechanism, improve the fault tolerance of the key-value cache, reduce the negative impact of abnormal key vectors on the model output quality, and improve the stability and reliability of model reasoning.

[0079] In some embodiments, the detection may further include value vector detection, which may be performed after determining the attention score of the current layer.

[0080] like Figure 2 As shown, step 150 determines the context output vector of the current layer according to the value vectors and attention scores of the multiple word units corresponding to the current layer, including:

[0081] Step 151: Detect the value vectors of multiple word units corresponding to the current layer to determine whether the value vectors contain any read errors caused by a storage medium failure.

[0082] Step 152: When it is detected that there are no reading errors in the value vectors of multiple word units corresponding to the current layer, the context output vector of the current layer is calculated based on the value vectors and attention scores of the multiple word units corresponding to the current layer.

[0083] Step 153: When a reading error is detected in the value vector of the second word among the multiple words corresponding to the current layer, the attention importance of the second word in the current layer is determined, and the context output vector of the current layer is corrected based on the attention importance of the second word in the current layer.

[0084] After calculating or correcting the attention score of the current layer, the value vector of each word element corresponding to the current layer is detected.

[0085] A storage medium failure of the key-value cache can easily lead to a read error in the value vector read from the key-value cache, such as a bit reversal error, which causes the sign bit inversion or the exponent key bit reversal in the floating-point number of the value vector, that is, the value vector read from the key-value cache is wrong. The wrong value vector will seriously interfere with the calculation of the attention mechanism and affect the model output results. Therefore, after reading the key-value vectors of multiple word elements corresponding to the current layer, this embodiment detects the value vectors of the multiple word elements read in the current layer. If there is no read error in the value vectors of the multiple word elements read in the current layer, the calculation of the context output vector of the current layer is performed normally; if there is a read error in the value vector of at least one word element read in the current layer, the context output vector of the current layer is corrected to avoid the abnormal value vector affecting the calculation of the attention mechanism, thereby avoiding affecting the model output results.

[0086] In some embodiments, the step 151 of detecting the value vectors of the multiple word-grams corresponding to the current layer includes:

[0087] Compare the value vectors of multiple word units corresponding to the current layer with the value vector threshold respectively;

[0088] If the value vectors of the multiple word units corresponding to the current layer do not exceed the value vector threshold, it is determined that the value vectors of the multiple word units corresponding to the current layer do not have a reading error;

[0089] If the value vector of the second word element corresponding to the current layer exceeds the value vector threshold, it is determined that there is a reading error in the value vector of the second word element corresponding to the current attention layer.

[0090] Wherein, the value vector threshold Threshold V Refers to the normal operation threshold of each element of the value vector. Value vector threshold Threshold V It can be set before actual inference is performed on the pre-trained language model. When the value vectors of the corresponding multiple word units are detected, the current layer Each element (or key bit, such as the index part) of the value vector of each corresponding word element and the value vector threshold Threshold V Compare. If the current layer Each element of the value vector of each corresponding word does not exceed the value vector threshold Threshold V , it indicates that the current layer The value vectors of the corresponding word units are running normally and there is no reading error; if the current layer At least one element of the value vector of at least one corresponding word-gram (such as the first word-gram) exceeds the value vector threshold Threshold V , it indicates that the current layer The value vector of the corresponding first word is abnormal, and there is a read error.

[0091] When the current layer is detected If there are no reading errors in the value vectors of the corresponding multiple word units, the current layer is executed normally. The calculation of the context output vector of the current layer does not require The context output vector is corrected.

[0092] For example, the current layer of the calculation or correction The attention score of the current layer is normalized by Softmax to obtain The corresponding attention weights of each word unit, for the current layer The corresponding attention weights of each word and the read value vector are weighted summed to calculate the current layer The context output vector of the current layer The context output vector of the next layer The context input vector of .

[0093] When the current layer is detected The corresponding second word token The value vector of V (l，j) In case of a reading error, the second word token In the current layer The importance of attention to the current layer The context output vector is corrected to avoid abnormal value vectors V (l，j) Affects the current layer The context output vector of .

[0094] In some embodiments, determining the attention importance of the second word-gram at the current layer includes:

[0095] Get the attention score of the second word corresponding to the current layer;

[0096] If the attention score of the second word corresponding to the current layer is less than the score threshold, it is determined that the attention importance of the second word in the current layer is less than the second importance threshold;

[0097] If the attention score of the second word corresponding to the current layer is greater than or equal to the score threshold, it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold.

[0098] Current layer The attention score of the current layer includes The attention score of each corresponding word. The corresponding second word token The higher the attention score of In the current layer The higher the importance of attention.

[0099] Set score threshold θ v . Set the current layer The corresponding second word token Attention score α j l With score threshold θ v Compare. If the current layer The corresponding second word token Attention score α j l Greater than or equal to score threshold θ v , then determine the second word token In the current layer The attention importance of is greater than or equal to the second importance threshold, that is, the second word token In the current layer The attention importance of The corresponding second word token Attention score α j l Less than the score threshold θ v , then determine the second word token In the current layer The attention importance of is less than the second importance threshold, that is, the second word token In the current layer The importance of attention is low.

[0100] This embodiment is based on the current layer The corresponding second word token Attention score α j l To accurately determine the second word token In the current layer The importance of attention in order to more effectively correct the current layer The context output vector of .

[0101] In some embodiments, modifying the context output vector of the current layer according to the attention importance of the second word in the current layer includes:

[0102] When it is determined that the attention importance of the second word-unit in the current layer is less than the second importance threshold, the value vector of the second word-unit corresponding to the current layer is discarded, and the context output vector of the current layer is calculated based on the value vectors and attention scores of other word-units corresponding to the current layer;

[0103] When it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold, the value vector of the second word corresponding to the current layer is reconstructed, the value vector of the second word in the key-value cache of the current layer is replaced with the reconstructed value vector, and the context output vector of the current layer is calculated based on the value vectors and attention scores of multiple words corresponding to the current layer.

[0104] If the second word token In the current layer The attention importance of is less than the second importance threshold, which means that the second word token In the current layer The attention importance of is low. Using the principle of attention sparsity, the second word token is directly discarded The value vector of , ignoring the second word token The contribution of the value vector of . When the context output vector is , the second word token The weighted summation term corresponding to the value vector of is set to zero vector or removed from the summation. The corresponding value vectors of other words do not have any reading errors and participate in the current layer normally. The calculation of the context output vector of the current layer The corresponding value vectors and attention scores of other words are weighted and summed to calculate the current layer The context output vector of the current layer The context output vector of the current layer after correction The context output vector of the next layer The context input vector of .

[0105] Because the second word token The value vector of is damaged, and its weight under the current attention mechanism is low, which has a negative impact on the current layer. The influence of the context output vector is small, so the second word token is directly discarded The value vector of avoids introducing erroneous information and has the lowest overhead.

[0106] In related technologies, attention sparsity is primarily used for proactive key-value cache pruning to improve efficiency. However, in this embodiment, attention sparsity is used for decision-making after an anomaly occurs. Specifically, the contribution of outlier vectors with low attention importance is directly discarded, applying attention sparsity to passive fault tolerance scenarios.

[0107] If the second word token In the current layer The attention importance of is greater than or equal to the second importance threshold, which means that the second word token In the current layer The attention importance of the second word token is high. The construction of the current context is very important and cannot be simply discarded. By reconstructing the current layer The corresponding second word token The value vector of , can get an approximate, available value vector. According to the current layer The corresponding value vectors of other words and the reconstructed second word token The value vector and attention score are weighted and summed to calculate the current layer The context output vector of the current layer The context output vector of the current layer after correction The context output vector of the next layer The context input vector of .

[0108] This embodiment uses low-rank recovery for important outlier vectors to ensure the retention of key information. At the same time, its backup storage and recovery I / O overhead is much lower than full-rank operations.

[0109] In addition, the current layer The second token in the key-value cache The value vector of is replaced by the reconstructed value vector to achieve the current layer Online repair of the key-value cache to ensure that the current layer is read again later The second token in the key-value cache When reconstructing the value vector of , the reconstructed, more accurate value vector is used. This prevents errors from continuing to propagate due to buffer reuse in subsequent decoding steps, which is particularly beneficial in scenarios such as long sequence generation.

[0110] In some embodiments, reconstructing the value vector of the second word-gram includes:

[0111] Obtain the low-rank value vector and reconstruction matrix of the second word corresponding to the current layer from the backup storage area;

[0112] Reconstruct the value vector of the second word corresponding to the current layer according to the low-rank value vector of the second word corresponding to the current layer and the reconstruction matrix.

[0113] Before the pre-trained language model performs actual reasoning, the low-rank value vector and reconstruction matrix of each word are stored in the backup storage area. When the value vector of The corresponding second word token The low-rank value vector of V lrj (Dimension 1× r ) and the current layer The reconstruction matrix . According to the current layer The corresponding second word token The low-rank value vector of V lrj and the current layer The reconstruction matrix , reconstruct the current layer The corresponding second word token The value vector of , the reconstructed value vector is: .

[0114] Low-rank decomposition in related technologies is mainly used for global compression, rather than on-demand, importance-based local recovery and repair. This embodiment uses low-rank recovery to reconstruct the second word token with acceptable I / O and computing overhead. The value vector is restored and the cache is repaired for subsequent use, improving the accuracy and reliability of the value vector. Moreover, the I / O overhead of loading the low-rank value vector and reconstructing the matrix is ​​much smaller than the overhead of loading or recalculating the full-rank value vector.

[0115] In some embodiments, before reading the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer of the multi-layer attention layer in step 110, the method further includes:

[0116] The value vectors of multiple word elements in the key-value cache of each layer are respectively used to form a value matrix;

[0117] Perform low-rank decomposition on the value matrix to obtain the projection matrix and reconstruction matrix;

[0118] According to the value matrix and the projection matrix, a low-rank matrix is ​​determined, where the low-rank matrix includes low-rank value vectors of multiple word units corresponding to each layer;

[0119] Store the reconstructed matrix and the low-rank matrix in the backup storage area.

[0120] Before the pre-trained language model performs inference on the input text, a representative task dataset can be selected. Based on the selected task dataset, multiple inferences are performed on the pre-trained language model. The key-value vectors generated at each layer during the inference process are stored in the corresponding key-value cache.

[0121] Collect the value vectors in the key-value cache and form a value matrix with the value vectors of all the word elements in each layer. l The value vectors of all word units constitute the value matrix For a specific layer l The value matrix of Perform a low-rank decomposition, such as singular value decomposition (SVD). Assume that the dimension of the value vector is d , select a value much smaller than d Rank r , the decomposition formula is:

[0122] .

[0123] According to the decomposition results, a specific layer is obtained l The projection matrix (Dimension d × r , usually by W r Export) and reconstruct the matrix (Dimension r × d , usually by or its approximate form). The reconstruction matrix Stored to backup storage area. Based on specific layer The projection matrix Sum Matrix , calculate the specific layer The low-rank matrix . The low-rank matrix V lr Store to backup storage. Specific layer The low-rank matrix V lr Include specific layers The corresponding low-rank value vector of each word, such as a specific layer The corresponding second word token The low-rank value vector of V lrj .

[0124] This embodiment pre-stores the low-rank value vectors and reconstruction matrices of each word element corresponding to each layer in a backup storage area, so that during the online inference process, the corresponding low-rank value vectors and reconstruction matrices can be directly obtained from the backup storage area to reconstruct abnormal value vectors, accelerate the online recovery process, and improve correction efficiency.

[0125] In some embodiments, before reading the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer of the multi-layer attention layer in step 110, the method further includes:

[0126] Select task dataset;

[0127] Based on the task dataset, a pre-trained language model is used to perform multiple inferences. The key-value vectors generated at each layer during the inference process are stored in the corresponding key-value cache.

[0128] Collect statistics on key vectors in the key-value cache and determine the key vector threshold;

[0129] Collect statistics on the value vectors in the key-value cache and determine the value vector threshold.

[0130] Before the pre-trained language model performs inference on the input text, a representative task dataset can be selected. Based on the selected task dataset, multiple inferences are performed on the pre-trained language model. The key-value vectors generated at each layer during the inference process are stored in the corresponding key-value cache.

[0131] The key vectors and value vectors in the key-value cache are collected and analyzed in detail, with a focus on the numerical distribution of floating-point numbers (such as FP16 / BF16), especially the bit patterns in the exponent part, because flipping these bits is most likely to cause significant changes in the numerical value.

[0132] Perform global or layer-by-layer statistics on all key vectors and all value vectors in all key-value caches, ignoring the differences in attention heads. Calculate the mean, standard deviation, specific percentile values ​​(such as the 99.99% percentile) of the key vector and value vector, and the range of outliers that may be generated after simulating bit flipping. Based on these calculated values, set the normal operation threshold of each element in the key vector, that is, the key vector threshold Threshold K , and set the normal operation threshold of each element in the value vector, that is, the value vector threshold Threshold V If the key vector exceeds the key vector threshold Threshold K , it indicates that the key vector has potential anomalies; if the value vector exceeds the value vector threshold Threshold V , it indicates that the value vector has potential anomalies.

[0133] This embodiment accurately determines the key vector threshold based on the key vector and value vector generated by multiple inferences on the pre-trained language model based on the selected task dataset. Threshold K Sum vector threshold Threshold V , to improve the detection accuracy of key vectors and value vectors.

[0134] The above is a coarse-grained statistics of key vectors and value vectors (ignoring attention heads). You can also perform fine-grained statistics on key vectors and value vectors (taking attention heads into account). For example, statistics are performed on the key vectors and value vectors in the key-value cache of each attention head in each layer. The mean, standard deviation, specific percentile values ​​(such as the 99.99% percentile) of the key vector and value vector are calculated separately, as well as the range of outliers that may be generated after simulating bit reversal. Based on these calculated values, the key vector threshold is set. Threshold K Sum vector threshold Threshold V .

[0135] It should be noted that the statistical granularity is different and the key vector threshold is set Threshold K Sum vector threshold Threshold V There will be some differences. This embodiment is compatible with different statistical granularities. The specific statistical granularity can be determined based on the computing and storage resources of the application scenario and the task accuracy during inference.

[0136] In some embodiments, each layer of the multi-layer attention layer includes multiple attention heads. When a read error is detected in the key vector of a first word-gram among multiple words-grams corresponding to a first attention head of a current layer, determining the attention importance of the first word-gram in the previous layer in step 140 and correcting the attention score of the current layer based on the attention importance of the first word-gram in the previous layer include:

[0137] Calculate the average attention score of each word corresponding to multiple attention heads in the previous layer;

[0138] Sort the average attention scores of multiple words corresponding to the previous layer in descending order;

[0139] If the average attention score of the first word corresponding to the previous layer ranks in the top target digit, then the attention importance of the first word in the previous layer is determined to be greater than or equal to the first importance threshold, and the key vector of the first word corresponding to the first attention head of the current layer is discarded. The attention score of the current layer is calculated based on the key vectors of the first words corresponding to other attention heads of the current layer, the key vectors of other words corresponding to multiple attention heads of the current layer, and the context input vector;

[0140] If the attention score of the first word corresponding to the previous layer is not ranked in the top target digits, it is determined that the attention importance of the first word in the previous layer is less than the first importance threshold, and the attention score of the previous layer is used as the attention score of the current layer.

[0141] Among them, when the attention importance of the first word in the previous layer is not determined, the key vector of the first word corresponding to the first attention head of the current layer is discarded, and the attention score of the current layer is calculated based on the key vectors of the first word corresponding to other attention heads of the current layer, the key vectors of other words corresponding to multiple attention heads of the current layer, and the context input vector.

[0142] When a read error is detected in a value vector of a second word-gram among multiple words-grams corresponding to a second attention head of a current layer, the steps of determining the attention importance of the second word-gram at the current layer and correcting the context output vector of the current layer according to the attention importance of the second word-gram at the current layer include:

[0143] Get the average attention score of the second word corresponding to multiple attention heads of the current layer;

[0144] If the average attention score of the second word-gram corresponding to the multiple attention heads of the current layer is less than the score threshold, it is determined that the attention importance of the second word-gram in the current layer is less than the second importance threshold, and the value vector of the second word-gram corresponding to the second attention head of the current layer is discarded. The context output vector of the current layer is calculated based on the value vectors of the second word-grams corresponding to the other attention heads of the current layer, the value vectors of the other word-grams corresponding to the multiple attention heads of the current layer, and the attention scores;

[0145] If the average attention score of the second word corresponding to the multiple attention heads of the current layer is greater than or equal to the score threshold, it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold, and the value vector of the second word corresponding to the second attention head of the current layer is reconstructed. The value vector of the second word in the key-value cache of the second attention head of the current layer is replaced with the reconstructed value vector, and the context output vector of the current layer is calculated based on the value vectors and attention scores of the multiple words corresponding to the multiple attention heads of the current layer.

[0146] Before actual inference is performed on the pre-trained language model, the value vectors of multiple word-units in the key-value cache of each attention head at each layer are combined into a value matrix. A low-rank decomposition is performed on the value matrix of each attention head at each layer to obtain a projection matrix and a reconstruction matrix for each attention head at each layer. Based on the value matrix and projection matrix of each attention head at each layer, a low-rank matrix is ​​determined. This low-rank matrix includes the low-rank vectors of multiple word-units corresponding to each attention head at each layer. The reconstruction matrix and low-rank matrix are stored in a backup storage area.

[0147] During the actual inference process of the pre-trained language model, the low-rank value vector and reconstruction matrix of the second word-unit corresponding to the second attention head of the current layer are obtained from the backup storage area. Based on the low-rank value vector and reconstruction matrix of the second word-unit corresponding to the second attention head of the current layer, the value vector of the second word-unit corresponding to the second attention head of the current layer is reconstructed.

[0148] This embodiment can perform more refined and targeted detection of the key vectors and value vectors of each word corresponding to each attention head of each attention layer, further improve the correction accuracy, effectively reduce the negative impact of abnormal key vectors and / or value vectors on the model output quality, and improve the stability and reliability of the inference results.

[0149] According to the fault tolerance method for key-value cache in the pre-trained language model provided in the embodiment of the present application, the numerical state of the key vector and value vector in the key-value cache is detected in real time. When there is a read error in the key vector and / or the value vector, the principles of inter-layer attention similarity, low rank and attention sparsity are used to perform targeted, low-overhead abnormal fault tolerance and cache repair. When there is a read error in the key vector in the key-value cache, the inter-layer attention similarity is used to refer to or directly reuse the attention score of the previous layer where no read error exists to avoid the negative impact of the abnormal key vector on the attention score of the current layer, so as to avoid complex response or redundant backup of the abnormal key vector, significantly reduce the fault tolerance overhead, and can effectively utilize computing and I / O resources. When a value vector in the key-value cache experiences a read error, its corresponding attention score is first evaluated. If the score is low, the outlier vector is discarded directly using the principle of attention sparsity to prevent error propagation. If the score is high, the low-rank value vector and reconstruction matrix of the outlier vector are loaded from the backup storage area at minimal I / O cost to reconstruct an approximate value vector. The reconstructed value vector is used not only for the current calculation but also for replacing the damaged full-rank value vector in the key-value cache, achieving online repair of the cache and ensuring a balance between efficiency and effectiveness of fault tolerance. Furthermore, this fault-tolerance mechanism can be integrated as a module into the pre-trained language model inference framework of related technologies. Its core concepts (such as decision-making based on previous-layer information and importance-aware recovery) can also provide reference for more advanced key-value cache management and fault-tolerance technologies in the future, with good compatibility and scalability.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0151] An embodiment of the present application also provides a computer program product.

[0152] Figure 3 A block diagram of a computer program product provided according to an embodiment of the present application.

[0153] like Figure 3 As shown, the computer program product 10 includes: a reading module 100 , a detection module 200 , a calculation module 300 , a correction module 400 and a determination module 500 .

[0154] A reading module 100 is configured to read, layer by layer, key-value vectors of multiple tokens related to the input text from a key-value cache of each layer of a multi-layer attention layer of a pre-trained language model;

[0155] A detection module 200 is configured to detect, when reading key-value vectors of multiple word-units corresponding to a current layer, whether the key vectors have a reading error caused by a storage medium failure;

[0156] A calculation module 300 is configured to calculate an attention score for the current layer based on the key vectors of the multiple word-grams corresponding to the current layer and the context input vector when detecting that the key vectors of the multiple word-grams corresponding to the current layer do not have any reading errors;

[0157] A correction module 400 is configured to, upon detecting a read error in a key vector of a first word-gram among a plurality of word-grams corresponding to a current layer, determine the attention importance of the first word-gram in a previous layer, and correct an attention score of the current layer based on the attention importance of the first word-gram in the previous layer;

[0158] Determination module 500 is used to determine the context output vector of the current layer based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is the context input vector of the next layer, and the context output vector of the last layer is used to generate the inference result corresponding to the input text.

[0159] In some embodiments, the correction module 400 includes a first correction unit and a second correction unit;

[0160] a first correction unit, configured to discard the key vector of the first word-gram corresponding to the current layer, if it is determined that the attention importance of the first word-gram in the previous layer is less than a first importance threshold, and calculate the attention score of the current layer based on the key vectors of other word-grams corresponding to the current layer and the context input vector;

[0161] The second correction unit is used to use the attention score of the previous layer as the attention score of the current layer when it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold.

[0162] In some embodiments, the correction module 400 further includes a sorting unit, a first determining unit, and a second determining unit;

[0163] The sorting unit is used to sort the attention scores of multiple words corresponding to the previous layer in descending order;

[0164] a first determining unit, configured to determine that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold if the attention score of the first word corresponding to the previous layer is ranked in the top target digit;

[0165] The second determining unit is used to determine that the attention importance of the first word in the previous layer is less than the first importance threshold if the attention score of the first word corresponding to the previous layer is not ranked in the top target digit.

[0166] In some embodiments, a score calculation module is further included;

[0167] A score calculation module is used to discard the key vector of the first word corresponding to the current layer when the attention importance of the first word in the previous layer is not determined, and calculate the attention score of the current layer based on the key vectors of other words corresponding to the current layer and the context input vector.

[0168] In some embodiments, the determination module 500 includes a detection unit, a calculation unit, and a third correction unit;

[0169] a detection unit, configured to detect value vectors of a plurality of word units corresponding to a current layer, and detect whether the value vectors have a read error caused by a storage medium failure;

[0170] a calculation unit, configured to calculate a context output vector of the current layer based on the value vectors and attention scores of the multiple word units corresponding to the current layer when detecting that none of the value vectors of the multiple word units corresponding to the current layer have any reading errors;

[0171] The third correction unit is used to determine the attention importance of the second word in the current layer when a reading error is detected in the value vector of the second word among the multiple words corresponding to the current layer, and to correct the context output vector of the current layer according to the attention importance of the second word in the current layer.

[0172] In some embodiments, the third correction unit includes a first correction sub-unit and a second correction sub-unit;

[0173] a first correction subunit, configured to discard the value vector of the second word-gram corresponding to the current layer when determining that the attention importance of the second word-gram in the current layer is less than a second importance threshold, and calculate the context output vector of the current layer based on the value vectors and attention scores of other word-grams corresponding to the current layer;

[0174] The second correction sub-unit is used to reconstruct the value vector of the second word corresponding to the current layer when it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold, replace the value vector of the second word in the key-value cache of the current layer with the reconstructed value vector, and calculate the context output vector of the current layer based on the value vectors and attention scores of multiple words corresponding to the current layer.

[0175] In some embodiments, the second correction subunit is further configured to:

[0176] Obtaining the low-rank value vector and reconstruction matrix of the second word corresponding to the current layer from the backup storage area;

[0177] Reconstruct the value vector of the second word corresponding to the current layer according to the low-rank value vector of the second word corresponding to the current layer and the reconstruction matrix.

[0178] In some embodiments, further comprising a composition module, a decomposition module, a vector determination module, and a storage module;

[0179] A construction module, used to construct a value matrix from the value vectors of multiple words in the key-value cache of each layer;

[0180] A decomposition module, configured to perform low-rank decomposition on the value matrix to obtain a projection matrix and a reconstruction matrix;

[0181] a vector determination module, configured to determine a low-rank matrix based on the value matrix and the projection matrix, wherein the low-rank matrix includes low-rank value vectors of multiple word units corresponding to each layer;

[0182] A storage module is used to store the reconstructed matrix and the low-rank matrix in the backup storage area.

[0183] In some embodiments, the third correction unit includes an acquisition subunit, a first determination subunit, and a second determination subunit;

[0184] Obtain a subunit for obtaining the attention score of the second word corresponding to the current layer;

[0185] a first determining subunit, configured to determine that the attention importance of the second word-gram in the current layer is less than a second importance threshold if the attention score of the second word-gram corresponding to the current layer is less than a score threshold;

[0186] The second determining subunit is used to determine that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold if the attention score of the second word corresponding to the current layer is greater than or equal to the score threshold.

[0187] In some embodiments, the detection module 200 includes a comparison unit, a third determination unit, and a fourth determination unit;

[0188] A comparison unit, configured to compare the key vectors of multiple word units corresponding to the current layer with the key vector threshold respectively;

[0189] a third determining unit, configured to determine that no read error exists in the key vectors of the multiple word units corresponding to the current layer if the key vectors of the multiple word units corresponding to the current layer do not exceed the key vector threshold;

[0190] a fourth determining unit, configured to determine that a read error exists in the key vector of the first word-unit corresponding to the current layer if the key vector of the first word-unit corresponding to the current layer exceeds the key vector threshold;

[0191] In some embodiments, the detection unit includes a comparison subunit, a third determination subunit, and a fourth determination subunit;

[0192] A comparison subunit, configured to compare the value vectors of multiple word units corresponding to the current layer with the value vector thresholds respectively;

[0193] a third determining subunit, configured to determine that none of the value vectors of the multiple word units corresponding to the current layer have any reading errors if none of the value vectors of the multiple word units corresponding to the current layer exceed the value vector threshold;

[0194] The fourth determination subunit is used to determine that there is a reading error in the value vector of the second word element corresponding to the current attention layer if the value vector of the second word element corresponding to the current layer exceeds the value vector threshold.

[0195] In some embodiments, the invention further includes a selection module, a reasoning module, a first statistics module and a second statistics module;

[0196] Selection module, used to select task data sets;

[0197] An inference module is used to perform multiple inferences based on the task dataset using a pre-trained language model, and the key-value vectors generated at each layer during the inference process are stored in a corresponding key-value cache;

[0198] a first statistics module, configured to collect statistics on key vectors in the key-value cache and determine the key vector threshold;

[0199] The second statistical module is used to perform statistics on the value vectors in the key-value cache and determine the value vector threshold.

[0200] For the description of the features in the embodiment corresponding to the computer program product 10, please refer to the relevant description of the embodiment corresponding to the fault-tolerant method of the key-value cache in the model, which will not be repeated here.

[0201] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the embodiment of the fault-tolerant method of key-value caching in any of the above models.

[0202] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of the fault-tolerant method embodiment of the key-value cache in any of the above models when running.

[0203] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0204] An embodiment of the present application further provides another computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the fault-tolerant method embodiment of the key-value cache in any of the above models.

[0205] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the fault-tolerant method embodiment of the key-value cache in any of the above models.

[0206] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0207] The above is a detailed introduction to the fault-tolerant method, product, device and medium of the key-value cache in a model provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core idea of ​​this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A fault-tolerant method for key-value caching in a model, characterized in that: include: Read the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer of the multi-layer attention layer of the pre-trained language model layer by layer; In the case of reading the key-value vectors of the multiple word-units corresponding to the current layer, detecting the key vectors of the multiple word-units corresponding to the current layer to detect whether the key vectors have a reading error caused by a storage medium failure; When it is detected that there are no reading errors in the key vectors of multiple word units corresponding to the current layer, the attention score of the current layer is calculated according to the key vectors of the multiple word units corresponding to the current layer and the context input vector; When a read error is detected in a key vector of a first word-gram among multiple words-grams corresponding to a current layer, determining the attention importance of the first word-gram in a previous layer, and correcting an attention score of the current layer according to the attention importance of the first word-gram in the previous layer; Determine the context output vector of the current layer based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is the context input vector of the next layer, and the context output vector of the last layer is used to generate the inference result corresponding to the input text; The step of correcting the attention score of the current layer according to the attention importance of the first word in the previous layer includes: If it is determined that the attention importance of the first word-gram in the previous layer is less than a first importance threshold, discard the key vector of the first word-gram corresponding to the current layer, and calculate the attention score of the current layer based on the key vectors of other words-grams corresponding to the current layer and the context input vector; When it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold, the attention score of the previous layer is used as the attention score of the current layer.

2. The fault-tolerant method for key-value caching in a model according to claim 1, characterized in that: Determining the attention importance of the first word in the previous layer includes: Sort the attention scores of multiple words corresponding to the previous layer in descending order; If the attention score of the first word corresponding to the previous layer is ranked in the top target digit, then it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold; If the attention score of the first word corresponding to the previous layer is not ranked in the top target digit, it is determined that the attention importance of the first word in the previous layer is less than the first importance threshold.

3. The fault-tolerant method for key-value caching in a model according to claim 1, characterized in that: In the case where a read error is detected in the key vector of the first word corresponding to the current layer, the method further includes: If the attention importance of the first word in the previous layer is not determined, the key vector of the first word corresponding to the current layer is discarded, and the attention score of the current layer is calculated based on the key vectors of other words corresponding to the current layer and the context input vector.

4. The fault-tolerant method for key-value caching in a model according to claim 1, characterized in that: The step of determining the context output vector of the current layer according to the value vectors and attention scores of the multiple word units corresponding to the current layer includes: Detecting the value vectors of multiple word elements corresponding to the current layer to determine whether the value vectors have a read error caused by a storage medium failure; When it is detected that the value vectors of multiple word units corresponding to the current layer do not have any reading errors, the context output vector of the current layer is calculated based on the value vectors and attention scores of the multiple word units corresponding to the current layer; When a reading error is detected in the value vector of the second word among the multiple words corresponding to the current layer, the attention importance of the second word in the current layer is determined, and the context output vector of the current layer is corrected based on the attention importance of the second word in the current layer.

5. The fault-tolerant method for key-value caching in a model according to claim 4, characterized in that: The modifying the context output vector of the current layer according to the attention importance of the second word in the current layer includes: If it is determined that the attention importance of the second word-gram in the current layer is less than the second importance threshold, discard the value vector of the second word-gram corresponding to the current layer, and calculate the context output vector of the current layer based on the value vectors and attention scores of other word-grams corresponding to the current layer; When it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold, the value vector of the second word corresponding to the current layer is reconstructed, the value vector of the second word in the key-value cache of the current layer is replaced with the reconstructed value vector, and the context output vector of the current layer is calculated based on the value vectors and attention scores of multiple words corresponding to the current layer.

6. The fault-tolerant method for key-value caching in a model according to claim 5, characterized in that: The reconstructing the value vector of the second word includes: Obtaining the low-rank value vector and reconstruction matrix of the second word corresponding to the current layer from the backup storage area; Reconstruct the value vector of the second word corresponding to the current layer according to the low-rank value vector of the second word corresponding to the current layer and the reconstruction matrix.

7. The fault-tolerant method for key-value caching in a model according to claim 6, characterized in that: Before reading the key-value vectors of multiple word units related to the input text in the key-value cache of each layer in the multi-layer attention layer of the pre-trained language model layer by layer, the method further includes: The value vectors of multiple word elements in the key-value cache of each layer are respectively used to form a value matrix; Performing low-rank decomposition on the value matrix to obtain a projection matrix and a reconstruction matrix; Determine a low-rank matrix according to the value matrix and the projection matrix, where the low-rank matrix includes low-rank value vectors of multiple word units corresponding to each layer; The reconstructed matrix and the low-rank matrix are stored in the backup storage area.

8. The fault-tolerant method for key-value caching in a model according to claim 5, characterized in that: Determining the attention importance of the second word at the current layer includes: Get the attention score of the second word corresponding to the current layer; If the attention score of the second word corresponding to the current layer is less than the score threshold, determining that the attention importance of the second word in the current layer is less than the second importance threshold; If the attention score of the second word corresponding to the current layer is greater than or equal to the score threshold, it is determined that the attention importance of the second word in the current layer is greater than or equal to the second importance threshold.

9. The fault-tolerant method for key-value caching in a model according to any one of claims 4 to 8, characterized in that: The detecting of the key vectors of the multiple word units corresponding to the current layer includes: Compare the key vectors of multiple word units corresponding to the current layer with the key vector threshold respectively; If the key vectors of the multiple word units corresponding to the current layer do not exceed the key vector threshold, it is determined that the key vectors of the multiple word units corresponding to the current layer do not have a read error; If the key vector of the first word element corresponding to the current layer exceeds the key vector threshold, it is determined that there is a read error in the key vector of the first word element corresponding to the current layer; The detecting of the value vectors of the multiple word units corresponding to the current layer includes: Compare the value vectors of multiple word units corresponding to the current layer with the value vector threshold respectively; If the value vectors of the multiple word units corresponding to the current layer do not exceed the value vector threshold, it is determined that the value vectors of the multiple word units corresponding to the current layer do not have a read error; If the value vector of the second word element corresponding to the current layer exceeds the value vector threshold, it is determined that there is a reading error in the value vector of the second word element corresponding to the current attention layer.

10. The fault-tolerant method for key-value caching in a model according to claim 9, characterized in that: Before reading the key-value vectors of multiple word units related to the input text in the key-value cache of each layer in the multi-layer attention layer of the pre-trained language model layer by layer, the method further includes: Select task dataset; Based on the task dataset, multiple inferences are performed using the pre-trained language model, and the key-value vectors generated at each layer during the inference process are stored in the corresponding key-value cache; Performing statistics on key vectors in the key-value cache to determine the key vector threshold; Statistics are collected on the value vectors in the key-value cache to determine the value vector threshold.

11. A computer program product, characterized in that include: The reading module is used to read the key-value vectors of multiple tokens related to the input text from the key-value cache of each layer in the multi-layer attention layer of the pre-trained language model layer by layer; a detection module, configured to detect, when reading the key-value vectors of the multiple word-units corresponding to the current layer, the key vectors of the multiple word-units corresponding to the current layer, to detect whether the key vectors have a reading error caused by a storage medium failure; A calculation module, configured to calculate an attention score of the current layer based on the key vectors of the multiple word units corresponding to the current layer and the context input vector when detecting that there are no reading errors in the key vectors of the multiple word units corresponding to the current layer; a correction module, configured to, upon detecting a read error in a key vector of a first word-gram among a plurality of words-grams corresponding to a current layer, determine the attention importance of the first word-gram in a previous layer, and correct an attention score of the current layer based on the attention importance of the first word-gram in the previous layer; A determination module is configured to determine a context output vector of the current layer based on the value vectors and attention scores of multiple word units corresponding to the current layer; wherein the context output vector of the current layer is used as the context input vector of the next layer, and the context output vector of the last layer is used to generate an inference result corresponding to the input text; The correction module includes: a first correction unit, configured to discard the key vector of the first word-gram corresponding to the current layer, if it is determined that the attention importance of the first word-gram in the previous layer is less than a first importance threshold, and calculate the attention score of the current layer based on the key vectors of other word-grams corresponding to the current layer and the context input vector; The second correction unit is used to use the attention score of the previous layer as the attention score of the current layer when it is determined that the attention importance of the first word in the previous layer is greater than or equal to the first importance threshold.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault-tolerant method for key-value caching in the model according to any one of claims 1 to 10 when executing the computer program.

13. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the fault-tolerant method for key-value caching in the model according to any one of claims 1 to 10 are implemented.

Images