Large model reasoning method and device, electronic equipment and storage medium

Abstract

The invention discloses a large model reasoning method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring long-term memory information required for reasoning to-be-processed information by a large model; sampling the long-term memory information based on an attention mechanism to obtain target information; and based on the target information, utilizing the large model to perform reasoning to obtain output information for the to-be-processed information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model reasoning method, device, electronic device and storage medium. Background Art

[0002] When a large model (LM) processes longer information, it may forget long-term information during the reasoning process, resulting in lower reasoning accuracy of the large model. Summary of the invention

[0003] In response to the above technical problems, the embodiments of the present application provide a large model reasoning method, device, electronic device and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a large model reasoning method, including:

[0006] Obtain the long-term memory information needed for the large model to reason about the information to be processed;

[0007] Sampling the long-term memory information based on an attention mechanism to obtain target information;

[0008] Based on the target information, the large model is used to infer and obtain output information for the information to be processed.

[0009] In some embodiments, obtaining the long-term memory information required for the large model to perform reasoning on the information to be processed includes:

[0010] Obtaining initial context information required by the large model to perform reasoning on the information to be processed;

[0011] The long-term memory information is determined from the initial context information; the long-term memory information is located in the word-grams other than the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information.

[0012] In some embodiments, sampling the long-term memory information based on the attention mechanism to obtain target information includes:

[0013] Determining an attention score of a word included in the long-term memory information;

[0014] Determine the first k words with the largest attention scores as the target information;

[0015] Wherein, k is the specified window length, which is a positive integer.

[0016] In some embodiments, the large model reasoning method further includes:

[0017] Determine the attention pool memory information and the short-term memory information from the initial context information; the attention pool memory information belongs to the long-term memory information and is located in the first second specific number of words in the first word sequence; the short-term memory information is located in the last first specific number of words in the first word sequence;

[0018] The step of obtaining output information for the information to be processed by using the large model reasoning based on the target information includes:

[0019] Based on the attention pool memory information, the target information and the short-term memory information, the large model is used to infer and obtain output information for the information to be processed;

[0020] Among them, the word elements corresponding to the target information are stored in the first storage location in sequence, the word elements corresponding to the short-term memory information are stored in the second storage location in sequence, and the attention pool memory information, the target information and the short-term memory information form the target context information.

[0021] In some embodiments, the method of obtaining output information for the information to be processed by using the large model reasoning based on the attention pool memory information, the target information and the short-term memory information includes:

[0022] In the process of using the large model to perform word-by-word reasoning on the output information, the target information stored in the first storage location and the short-term memory information stored in the second storage location are updated based on the attention mechanism and the word-units inferred by the large model;

[0023] Based on the attention pool memory information, the target information stored in the updated first storage location, and the short-term memory information stored in the updated second storage location, the large model is used to continue reasoning about the next word on the output information.

[0024] In some embodiments, in the process of using the large model to perform word-by-word reasoning on the output information, updating the target information stored in the first storage location and the short-term memory information stored in the second storage location based on the attention mechanism and the word-by-word reasoning obtained by the large model includes:

[0025] In the case where it is determined that any first target word in the second word sequence corresponding to the output information has been inferred using the large model, a second target word in the word corresponding to the target information currently stored in the first storage location is determined based on the attention mechanism; the second target word is the word with the smallest attention score in the word corresponding to the target information currently stored in the first storage location;

[0026] After deleting the second target word from the first storage location, the word stored after the second target word in the first storage location is sequentially moved forward;

[0027] After the first word element in the word element corresponding to the short-term memory information currently stored in the second storage position is moved to the end position of the first storage position, the word elements stored after the first word element in the second storage position are sequentially moved forward;

[0028] The first target word-gram is stored at the end position of the second storage location.

[0029] In some embodiments, determining, based on the attention mechanism, a second target word-gram in the word-gram corresponding to the target information currently stored in the first storage location includes:

[0030] Based on the attention mechanism, determining an attention score of each word corresponding to the target information currently stored in the first storage location;

[0031] Based on the binary search method, the second target word-gram with the smallest attention score is determined from the word-grams corresponding to the target information currently stored in the first storage location.

[0032] In a second aspect, an embodiment of the present application provides a large model reasoning device, including:

[0033] An acquisition module is used to obtain the long-term memory information required by the large model to reason about the information to be processed;

[0034] A sampling module, used for sampling the long-term memory information based on an attention mechanism to obtain target information;

[0035] The reasoning module is used to obtain output information for the information to be processed based on the target information and using the large model for reasoning.

[0036] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store executable data instructions; when the processor is used to execute the executable data instructions stored in the memory, the steps in the large model reasoning method described in the first aspect are implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the large model reasoning method described in the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 One of the flowcharts of a large model reasoning method provided in an embodiment of the present application;

[0040] Figure 2 A second flowchart of a large model reasoning method provided in an embodiment of the present application;

[0041] Figure 3 A third flowchart of a large model reasoning method provided in an embodiment of the present application;

[0042] Figure 4 A fourth flowchart of a large model reasoning method provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a long short-term memory strategy for a large model provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of long-term memory update of a large model provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of the structure of a large model reasoning device provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0048] It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0049] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant technical knowledge is first introduced as follows.

[0050] When a large model processes long context information (for example, length L), the traditional attention mechanism needs to compare the attention score of each position of the context information with all other positions, which leads to O(L 2 ). In the case of a long context, this calculation method will occupy a large amount of video memory resources, resulting in video memory overflow in the reasoning process of large models. In order to solve this problem, the relevant technology introduces the key-value state value cache (Key-ValueCache, KV Cache) mechanism and sliding window strategy, and introduces the attention pool information (Attention Sink) as a prompt word, which can avoid the occurrence of video memory overflow to a certain extent and improve the reasoning performance of large models. However, the relevant technology is limited by the size of the context window, and the large model will forget the historical context information during the reasoning process. Although the introduction of Attention Sink can alleviate this problem to a certain extent, it still cannot effectively retain the important content of the historical context information, resulting in low reasoning accuracy of large models.

[0051] In order to overcome at least some of the above-mentioned defects existing in the related art, the embodiments of the present application provide a large model reasoning method, device, electronic device and storage medium, by sampling the long-term memory information required for the large model to reason about the information to be processed based on the attention mechanism, obtaining the more important target information in the long-term memory information, and then based on the target information, using the large model reasoning to obtain the output information for the information to be processed, which can improve the phenomenon of forgetting important long-term memory information in the large model reasoning process and improve the reasoning accuracy of the large model.

[0052] The following is an illustrative introduction to the large model reasoning method, device, electronic device and storage medium provided in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.

[0053] Figure 1 One of the flow charts of the large model reasoning method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0054] S101, obtaining the long-term memory information required by the large model to reason about the information to be processed.

[0055] It should be noted that the large model in the embodiments of the present application can be any machine learning or deep learning model with large parameters, which is a generative model. For example, a large language model (LLM, used to process and understand human language, perform tasks such as text generation, translation and sentiment analysis), a diffusion model (used to perform tasks such as image generation), etc.

[0056] It should be noted that the information to be processed in the embodiment of the present application may be data or content that the large model needs to analyze, understand or generate. The information to be processed includes but is not limited to text information, image information, audio information and video information.

[0057] It should be noted that the long-term memory information in the embodiments of the present application can be long-term information with accumulation and association used to support large models to perform reasoning or generation tasks. Long-term memory information includes but is not limited to text information, image information, audio information, video information, etc.

[0058] In some embodiments, the long-term memory information required for the large model to reason about the information to be processed can be obtained from a memory module plug-in of the large model.

[0059] In some embodiments, the long-term memory information required by the large model to reason about the information to be processed can be obtained from the task information that has been processed by the large model.

[0060] S102. Sampling the long-term memory information based on the attention mechanism to obtain target information.

[0061] It should be noted that the core of the attention mechanism is to assign different weights or attention scores to information at different positions in order to obtain the key information required for the current processing task or the current reasoning task.

[0062] In an embodiment of the present application, the long-term memory information required for the large model to reason about the information to be processed can be sampled based on the attention mechanism to obtain relatively important or critical target information required for the large model to reason about the information to be processed.

[0063] In some embodiments, based on the attention mechanism, different weights or attention scores can be assigned to information at different positions in the long-term memory information, and then the target information can be determined based on the weights or attention scores.

[0064] In some embodiments, information in the long-term memory information that is assigned a higher weight or attention score (eg, the assigned weight or attention score is greater than a specific value) may be determined as target information.

[0065] S103: Based on the target information, use the large model to infer and obtain output information for the information to be processed.

[0066] In some embodiments, based on target information sampled from long-term memory information, a large model can be used to perform token-by-token reasoning to obtain output information for the information to be processed.

[0067] In some embodiments, the target information and the information to be processed can be preprocessed (for example, word segmentation, stop word removal, etc.), and then the preprocessed target information and the information to be processed can be spliced ​​together and input into the large model, so that the large model can infer the output information for the information to be processed based on the target information.

[0068] It can be understood that the large model inference method provided in the embodiment of the present application obtains relatively important target information in the long-term memory information by sampling the long-term memory information required for the large model to reason about the information to be processed based on the attention mechanism, and then based on the target information, uses the large model to reason to obtain the output information for the information to be processed, which can reduce storage occupancy, improve the phenomenon of forgetting important long-term memory information in the large model reasoning process, and improve the reasoning accuracy of the large model.

[0069] In some embodiments, obtaining the long-term memory information required for the large model to perform reasoning on the information to be processed includes:

[0070] Obtaining initial context information required by the large model to perform reasoning on the information to be processed;

[0071] The long-term memory information is determined from the initial context information; the long-term memory information is located in the word-grams other than the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information.

[0072] For example, in a text processing task, the previous and subsequent information of the text is crucial for the large model to understand the meaning and intention of the current text. Context information can include background information (background knowledge of the current reasoning task, such as the topic of the conversation, the user's intention, etc.), historical interaction information, and short-term interaction information. It should be noted that in the embodiment of the present application, the context information before sampling the long-term memory information is referred to as initial context information.

[0073] It should be noted that in the process of the large model performing token-by-token reasoning on the information to be processed, contextual information will gradually accumulate and increase.

[0074] In some embodiments, the context information required for the large model to reason about the information to be processed can be obtained from a memory module plug-in of the large model.

[0075] In some embodiments, the context information required by the large model to reason about the information to be processed can be obtained from the task information that has been processed by the large model.

[0076] In some embodiments, the word-grams except for the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information can be used as long-term memory information required for the large model to perform reasoning on the information to be processed.

[0077] It should be noted that each word in the first word sequence is the smallest basic unit obtained after the initial context information is split and processed.

[0078] It should be noted that the first specific number in the embodiment of the present application can be adaptively set based on actual applications, and the embodiment of the present application does not make any specific limitation on this.

[0079] For example, Figure 2 The second flow chart of a large model reasoning method provided in the embodiment of the present application is as follows: Figure 2 As shown, the method includes:

[0080] S201, obtaining the initial context information required by the large model to perform reasoning on the information to be processed.

[0081] S202, determining long-term memory information from the initial context information; the long-term memory information is located in word-grams other than a first specific number of word-grams in a first word-gram sequence corresponding to the initial context information.

[0082] S203. Sampling the long-term memory information based on the attention mechanism to obtain target information.

[0083] S204: Based on the target information, use the large model to infer and obtain output information for the information to be processed.

[0084] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0085] It can be understood that the embodiment of the present application determines the scope of long-term memory information by using the words other than the last first specific number of words in the first word sequence corresponding to the initial context information as the long-term memory information required for the large model to reason about the information to be processed. The large model can reduce unnecessary calculations during the reasoning process, which helps to speed up the reasoning speed of the large model.

[0086] In some embodiments, sampling the long-term memory information based on the attention mechanism to obtain target information includes:

[0087] Determining an attention score of a word included in the long-term memory information;

[0088] Determine the first k words with the largest attention scores as the target information;

[0089] Wherein, k is the specified window length, which is a positive integer.

[0090] In an embodiment of the present application, the attention score of each word included in the long-term memory information can be determined based on the attention mechanism, and then the top k words with the largest attention scores are determined as the target information sampled from the long-term memory information.

[0091] It can be understood that the larger the attention score is, the higher the importance of the word in the current long-term memory information is.

[0092] It should be noted that k in the embodiment of the present application is a designated window length for sampling to obtain target information, which can be adaptively set based on actual applications, and the embodiment of the present application does not specifically limit this. For example, k is 5, 10, or 15.

[0093] In some embodiments, an attention map corresponding to the long-term memory information can be obtained, and the attention scores of the words included in the long-term memory information can be determined from the attention map. Among them, the attention map reveals the attention of the large model to each input token when generating each output token. A high attention score indicates that the large model is highly dependent on the input token at the corresponding position when generating the current token.

[0094] In some embodiments, the word-unit included in the long-term memory information is represented by a vector, and the vector representation includes a query vector (Q), a key vector (K), and a value vector (V). When calculating the attention score of a word-unit, the query vector of the word-unit can be dot-producted with the key vectors of other word-units to obtain the attention score of the word-unit.

[0095] For example, Figure 3 The third flow chart of a large model reasoning method provided in the embodiment of the present application is as follows: Figure 3 As shown, the method includes:

[0096] S301, obtaining the initial context information required by the large model to perform reasoning on the information to be processed.

[0097] S302, determining long-term memory information from the initial context information; the long-term memory information is located in word-grams other than a first specific number of word-grams in a first word-gram sequence corresponding to the initial context information.

[0098] S303: Determine the attention score of the word-unit included in the long-term memory information.

[0099] S304: Determine the first k words with the largest attention scores as target information.

[0100] S305: Based on the target information, use the large model to infer and obtain output information for the information to be processed.

[0101] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0102] It can be understood that the embodiment of the present application determines the top k word-grams with the largest attention scores in the long-term memory information as the sampled target information. Since the top k word-grams with the highest attention scores often contain important information, this plays a key role in the large model's reasoning about the information to be processed, and helps to improve the reasoning accuracy of the large model.

[0103] In some embodiments, the large model reasoning method further includes:

[0104] Determine the attention pool memory information and the short-term memory information from the initial context information; the attention pool memory information belongs to the long-term memory information and is located in the first second specific number of words in the first word sequence; the short-term memory information is located in the last first specific number of words in the first word sequence;

[0105] The step of obtaining output information for the information to be processed by using the large model reasoning based on the target information includes:

[0106] Based on the attention pool memory information, the target information and the short-term memory information, the large model is used to infer and obtain output information for the information to be processed;

[0107] Among them, the word elements corresponding to the target information are stored in the first storage location in sequence, the word elements corresponding to the short-term memory information are stored in the second storage location in sequence, and the attention pool memory information, the target information and the short-term memory information form the target context information.

[0108] In the embodiment of the present application, after obtaining the initial context information required by the large model for reasoning about the information to be processed, the first second specific number of words in the first word sequence corresponding to the initial context information are used as the attention pool memory information, the last first specific number of words in the first word sequence are used as the short-term memory information, and the remaining words in the first word sequence are the long-term memory information. Since the long-term memory information is relatively large, the long-term memory information can be sampled based on the attention mechanism to obtain more important or more critical target information in the long-term memory information, and then based on the attention pool memory information, the more important or more critical target information in the long-term memory information, and the short-term memory information, the large model is used for reasoning to obtain the output information for the information to be processed.

[0109] It should be noted that in large model reasoning, the beginning of the context information (i.e., the initial token) usually attracts a lot of attention. When processing long texts or complex conversations, the beginning of the context information helps the large model capture key clues in the context and ensure the coherence and accuracy of reasoning. In addition, in multi-round conversation scenarios, short-term memory information is crucial to maintaining the coherence of the conversation. The large model needs to remember the recent conversation content in order to better understand the current question and give an appropriate answer.

[0110] In an embodiment of the present application, the first second specific number of words in the first word sequence corresponding to the initial context information are used as attention pool memory information, the last first specific number of words in the first word sequence are used as short-term memory information, and the remaining words in the first word sequence are used as long-term memory information. The long-term memory information is sampled to obtain more important or more critical target information in the long-term memory information. Finally, the attention pool memory information, the more important or more critical target information in the long-term memory information, and the short-term memory information are combined, which helps the large model to perform more comprehensive and accurate reasoning on the information to be processed, thereby further improving the reasoning accuracy of the large model.

[0111] It should be noted that the second specific number in the embodiment of the present application can be adaptively set based on actual applications, and the embodiment of the present application does not specifically limit this. For example, the first two words in the first word-word sequence corresponding to the initial context information are used as the attention pool memory information required for the large model to reason about the information to be processed, or the first five words in the first word-word sequence corresponding to the initial context information are used as the attention pool memory information required for the large model to reason about the information to be processed.

[0112] It can be understood that the attention pool memory information belongs to the long-term memory information and plays the most important or critical role in large model reasoning.

[0113] For example, Figure 4A fourth flow chart of a large model reasoning method provided in an embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0114] S401, obtaining the initial context information required by the large model to perform reasoning on the information to be processed.

[0115] S402. Determine long-term memory information, attention pool memory information and short-term memory information from the initial context information; the long-term memory information is located in the word-grams other than the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information; the attention pool memory information belongs to the long-term memory information and is located before the second specific number of word-grams in the first word-gram sequence; the short-term memory information is located after the first specific number of word-grams in the first word-gram sequence.

[0116] S403: Determine the attention score of the word-unit included in the long-term memory information.

[0117] S404: Determine the first k words with the largest attention scores as target information.

[0118] S405. Based on the attention pool memory information, the target information and the short-term memory information, the large model is used to infer and obtain output information for the information to be processed.

[0119] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0120] In some embodiments, the word units corresponding to the target information are sequentially stored in the first storage location, and the word units corresponding to the short-term memory information are sequentially stored in the second storage location. Moreover, the first storage location and the second storage location have fixed storage capacities.

[0121] In some embodiments, the storage capacity of the first storage location may be determined based on the number of word units corresponding to the target information, and the storage capacity of the second storage location may be determined based on the number of word units corresponding to the short-term memory information.

[0122] In some embodiments, the key-value vector or key-value state value of the word element corresponding to the target information can be stored in a first storage location, and the key-value vector or key-value state value of the word element corresponding to the short-term memory information can be stored in a second storage location.

[0123] It should be noted that the large model performs reasoning based on the attention pool memory information, target information and short-term memory information. It can be understood that the large model is essentially based on the key-value vector or key-value state value of the word unit corresponding to the attention pool memory information, the key-value vector or key-value state value of the word unit corresponding to the target information, and the key-value vector or key-value state value of the word unit corresponding to the short-term memory information.

[0124] It should be noted that the attention pool memory information, target information and short-term memory information can form target context information, so that the large model can infer the output information for the information to be processed based on the target context information.

[0125] It can be understood that the embodiment of the present application forms target context information by combining the attention pool memory information, the more important or critical target information in the long-term memory information, and the short-term memory information, so that the large model can infer the output information for the information to be processed based on the target context information, which helps the large model to perform more comprehensive and accurate reasoning on the information to be processed, and further improves the reasoning accuracy of the large model.

[0126] In some embodiments, the method of obtaining output information for the information to be processed by using the large model reasoning based on the attention pool memory information, the target information and the short-term memory information includes:

[0127] In the process of using the large model to perform word-by-word reasoning on the output information, if the information in the storage location reaches the storage capacity limit, the attention mechanism and the word-grams inferred by the large model update the target information stored in the first storage location and the short-term memory information stored in the second storage location;

[0128] Based on the attention pool memory information, the target information stored in the updated first storage location, and the short-term memory information stored in the updated second storage location, the large model is used to continue reasoning about the next word on the output information.

[0129] It should be noted that when the large model infers the information to be processed, it infers word by word to obtain the output information. In the process of word by word inference of the large model, the short-term memory information stored in the second storage location of the fixed length will change in real time according to the word that the large model has inferred, and the long-term memory information will gradually accumulate and increase, and the target information stored in the first storage location of the fixed length sampled from the gradually accumulated long-term memory information will also change accordingly.

[0130] In an embodiment of the present application, in the process of using a large model to perform word-by-word reasoning on the output information, the target information stored in the first storage location and the short-term memory information stored in the second storage location can be updated based on the attention mechanism and the word-by-word reasoning obtained by the large model, and then based on the attention pool memory information, the updated target information stored in the first storage location and the updated short-term memory information stored in the second storage location, the large model is used to continue reasoning about the next word-by-word on the output information; the updating process of the target information and the short-term memory information and the word-by-word reasoning process of the large model are executed in a loop until the large model obtains complete output information by word-by-word reasoning.

[0131] It can be understood that the embodiments of the present application combine the attention mechanism with the target information in the long-term memory information and the storage location update mechanism of the short-term memory information, so that the large model can focus more on the information that is more critical to the current reasoning task during the reasoning process.

[0132] In some embodiments, in the process of using the large model to perform word-by-word reasoning on the output information, updating the target information stored in the first storage location and the short-term memory information stored in the second storage location based on the attention mechanism and the word-by-word reasoning obtained by the large model includes:

[0133] In the case where it is determined that any first target word in the second word sequence corresponding to the output information has been inferred using the large model, a second target word in the word corresponding to the target information currently stored in the first storage location is determined based on the attention mechanism; the second target word is the word with the smallest attention score in the word corresponding to the target information currently stored in the first storage location;

[0134] After deleting the second target word from the first storage location, the word stored after the second target word in the first storage location is sequentially moved forward;

[0135] After the first word element in the word element corresponding to the short-term memory information currently stored in the second storage position is moved to the end position of the first storage position, the word elements stored after the first word element in the second storage position are sequentially moved forward;

[0136] The first target word-gram is stored at the end position of the second storage location.

[0137] In an embodiment of the present application, when the large model has inferred a new word (i.e., any first target word in the second word sequence corresponding to the output information), the word with the smallest attention score in the word corresponding to the target information currently stored in the first storage position can be deleted based on the attention mechanism, and then the subsequent word in the first storage position is moved forward one position in sequence, and then the first word in the word corresponding to the short-term memory information currently stored in the second storage position is migrated to the end position of the first storage position, and the subsequent word in the second storage position is moved forward one position in sequence, and finally the word newly inferred by the large model is stored in the end position of the second storage position. At this time, an update process of the target information stored in the first storage position and the short-term memory information stored in the second storage position is completed, so that the large model can continue to infer the next word based on the updated target information stored in the first storage position and the updated short-term memory information stored in the second storage position, and in combination with the attention pool memory information.

[0138] It should be noted that the word with the smallest attention score currently stored in the first storage location is a relatively unimportant word of the large model in the current reasoning task, so it can be deleted.

[0139] It can be understood that the embodiment of the present application uses a specific streaming storage position update mechanism. When calculating the attention score based on the attention mechanism, it is only necessary to calculate the attention score of the word corresponding to the target information stored in the first storage position of a fixed length, delete the word with the smallest attention score, and then move the positions of the subsequent words forward one by one. There is no need to calculate the word with a higher attention score as the target information from the huge long-term memory information based on the attention mechanism. Therefore, the embodiment of the present application can improve the reasoning efficiency and speed of large models.

[0140] In some embodiments, determining, based on the attention mechanism, a second target word-gram in the word-gram corresponding to the target information currently stored in the first storage location includes:

[0141] Based on the attention mechanism, determining an attention score of each word corresponding to the target information currently stored in the first storage location;

[0142] Based on the binary search method, the second target word-gram with the smallest attention score is determined from the word-grams corresponding to the target information currently stored in the first storage location.

[0143] It should be noted that binary search is an efficient search algorithm that divides the ordered array into two halves and gradually narrows the search range, so that the target element can be quickly located.

[0144] In an embodiment of the present application, the attention scores of each word corresponding to the target information currently stored in the first storage location can be first determined based on the attention mechanism, and then based on the binary search method, the word with the smallest attention score can be found from the attention scores of each word corresponding to the target information, which is the second target word.

[0145] It can be understood that the embodiment of the present application can improve the efficiency of locating the word with the smallest attention score by searching for the word with the smallest attention score from the attention scores of each word corresponding to the target information based on the binary search method.

[0146] The large model reasoning method proposed in this application is introduced below through a specific embodiment. It mainly includes the following steps 1 and 2:

[0147] Step 1: Obtain the key (Key) value (Value) state value (hereinafter referred to as KV state value) of the long-term memory information and the KV state value of the short-term memory information required by the large model to reason about the information to be processed.

[0148] In an embodiment of the present application, when the large model performs inference token by token, the KV state value storage location long-short-term memory strategy is adopted to divide the KV state value into three parts and save them in the storage location space. The three parts are an Attention Sink module for storing Attention Sink information (corresponding to the attention pool memory information in the above text), a long-term memory module for storing long-term memory information, and a short-term memory module for storing short-term memory information.

[0149] When the input sequence length is L (the number of tokens in the input sequence) of long context information, the AttentionSink module completely saves the KV state values ​​of the tokens at the set number of positions before the input sequence (for example, the first position or the first two positions); the short-term memory module completely saves the KV state values ​​of the tokens near the current reasoning position of the input sequence (i-Ls to i, i is the token number of the current reasoning position, and Ls is the window length of the short-term memory module); the long-term memory module samples the state values ​​at the middle position of the KV state value cache according to the attention score in the attention map, and selects the first k KV state values ​​with higher attention scores to save. Where k is the window length of the long-term memory module, which is a positive integer.

[0150] In some embodiments, when the query vector q of the i-th (1<i≤L) token in the input sequence of length L is i ∈R 1×d (d represents the query vector dimension), it will be combined with the key vector K∈R L×dPerform a dot product operation to calculate the attention score. The calculation of the attention score is shown in formula (1):

[0151] Attention Score = softmax(q i K T )(1)

[0152] Among them, Attention Score represents the attention score; softmax represents the normalization function, which is used to normalize the attention score to obtain the final attention score.

[0153] In some embodiments, based on the attention score, the long-term memory module samples the KV state values ​​corresponding to the positions of the k elements with the highest attention scores from the KV state value cache corresponding to the long-term memory information in the long context information according to the window length k, and stores them sequentially in the long-term memory module.

[0154] In some embodiments, during the large model inference process, the KV state values ​​in the Attention Sink module, the long-term memory module, and the short-term memory module can be concatenated to form a new KV state value cache, where the key cache and the value cache are K∈R i×d ,V∈R i×d (i represents the number of the token currently being inferred). In addition, the KV state value cache will also be associated with the vector x corresponding to the next token. i+1 The calculated K i+1 ,V i+1 Splice to get the new KV state value cache K∈R (i +1)×d ,V∈R (i+1)×d , thereby executing the reasoning task of the next token.

[0155] For example, Figure 5 A schematic diagram of a long-short-term memory strategy for a large model provided in an embodiment of the present application. Figure 5 As shown in the figure, in the attention map, different grid fillings represent different values ​​after the dot product of the query vector Q and the key vector K. The higher the value, the higher the attention level of the token. Based on the attention map, the attention pool memory information, the fixed-length long-term memory information, and the short-term memory information are determined. Figure 5The attention pool memory information in is the KV state value (k1, v1) of the first token, the fixed-length long-term memory information is the KV state values ​​(k2, v2; k3, v3; k4, v4; k5, v5; k6, v6) of the second to sixth tokens, and the short-term memory information is the KV state values ​​(k7, v7; k8, v8) of the seventh and eighth tokens. In addition, the key value state values ​​(k2, v2; k4, v4) of the first two tokens with the highest attention scores sampled from the long-term memory information based on the attention score are used as important information in the long-term memory information.

[0156] It is understandable that the embodiment of the present application proposes a KV state value cache long-term and short-term memory strategy, which improves the catastrophic information forgetting problem in the large model reasoning process by sampling and extracting important information from the long-term memory information in the KV state value cache, and combining the attention pool memory information and short-term memory information. By adopting the KV state value cache long-term and short-term memory strategy, the large model can maintain the memory of long-term span context information and improve its reasoning accuracy in a long context environment.

[0157] Step 2: Streaming update of long-term memory information.

[0158] It should be noted that when the length of long-term memory information L l When the window length k of the long-term memory module is much larger than that of the long-term memory module, the time complexity of the sampling algorithm for long-term memory information is O(klogL l ), a long-term memory module with a fixed length can be used to record the KV state value of the long-term memory information, so that the time complexity of the sampling algorithm operation is reduced to O(logk). At this time, the large model has a higher reasoning speed.

[0159] In some embodiments, the process of streaming long-term memory information includes:

[0160] Step 2.1: Encode input data and initialize KV cache.

[0161] In some embodiments, the original input data can be preprocessed, and the preprocessed word segmentation results can be converted into word embeddings or feature vectors using a pretrained embedding model; during the first forward propagation of the large model, the KV state value cache is initialized for each layer of the large model and the intermediate state values ​​of the KV state value cache are saved.

[0162] Step 2.2: Determine the length of long-term memory information.

[0163] In some embodiments, during the token-by-token reasoning process, it can be determined whether the length of the long-term memory information in the current KV state value cache reaches the window length k of the long-term memory module; if the length of the long-term memory information does not exceed k, new data continues to be added to the long-term memory module; if the length of the long-term memory information exceeds k, step 2.3 is executed for streaming update.

[0164] Step 2.3: Streaming updates the information in the long-term memory module.

[0165] In some embodiments, all KV state values ​​stored in the long-term memory module can be traversed, the attention score of the token corresponding to each KV state value can be calculated, the KV state value corresponding to the token with the lowest attention score is found and deleted, and the KV state values ​​after the KV state value corresponding to the token with the lowest attention score are migrated forward; then the current first KV state value stored in the short-term memory module is migrated to the end of the long-term memory module; and then other KV state values ​​in the short-term memory module are migrated forward; finally, the KV state value of the latest token generated during the reasoning process is stored at the end of the short-term memory module.

[0166] Step 2.4: Repeat steps 2.2 and 2.3 until all output information inferences for the input sequence are completed.

[0167] For example, Figure 6 A schematic diagram of long-term memory update of a large model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown in , the fixed length of the long-term memory information is 7, where the newly added KV state value is the KV state value migrated from the short-term memory module. When the large model infers a new token, the KV state value of the first token currently stored in the short-term memory module needs to be migrated to the end of the long-term memory module so that the KV state value of the newly inferred token can be added to the end of the short-term memory module. At this time, for the long-term memory module, you can find the KV state value corresponding to the token with the lowest attention score and delete it, and migrate the KV state value after the KV state value corresponding to the token with the lowest attention score forward, and then migrate the first KV state value currently stored in the short-term memory module to the end of the long-term memory module. As shown in Figure 7 In the example, after deleting the KV status value of the 5th token with the lowest attention score in the long-term memory module, the KV status values ​​of the 6th and 7th tokens are moved forward one position respectively, and then the newly added KV status values ​​are stored at the end of the long-term memory module. At this time, the length of the updated long-term memory information is still 7, and the time complexity of the sampling algorithm is reduced.

[0168] It is understandable that the embodiment of the present application proposes a long-term memory information streaming update strategy, which optimizes the operation speed of the sampling extraction algorithm in a long context environment by setting a fixed-length KV state value cache and replacing the corresponding KV state value in the KV state value cache according to the attention score during the large model reasoning process. The fixed-length KV state value cache mechanism can reduce the computational complexity of the sampling algorithm from O(klogL l ) is reduced to O(logk), where L l is the length of the long-term memory information, k is the fixed window length of the long-term memory module, and k<<L l , effectively improving the reasoning speed of large models in long context environments.

[0169] The large model reasoning device provided in the embodiment of the present application is described below. The large model reasoning device described below and the large model reasoning method described above can be referenced to each other.

[0170] Figure 7 A schematic diagram of the structure of a large model reasoning device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device includes: an acquisition module 710, a sampling module 720 and an inference module 730; wherein:

[0171] An acquisition module 710 is used to obtain long-term memory information required by the large model to perform reasoning on the information to be processed;

[0172] A sampling module 720 is used to sample the long-term memory information based on an attention mechanism to obtain target information;

[0173] The reasoning module 730 is used to obtain output information for the information to be processed based on the target information and using the large model for reasoning.

[0174] The large model inference device provided in the embodiment of the present application samples the long-term memory information required for the large model to reason about the information to be processed based on the attention mechanism, obtains the more important target information in the long-term memory information, and then based on the target information, uses the large model to reason to obtain the output information for the information to be processed, which can improve the phenomenon of forgetting important long-term memory information in the large model reasoning process and improve the reasoning accuracy of the large model.

[0175] In some embodiments, the obtaining module 710 includes:

[0176] An obtaining unit, used for obtaining initial context information required by the large model to perform reasoning on the information to be processed;

[0177] The first determining unit is used to determine the long-term memory information from the initial context information; the long-term memory information is located in the word-grams other than the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information.

[0178] In some embodiments, the sampling module 720 includes:

[0179] A second determining unit, configured to determine an attention score of a word included in the long-term memory information;

[0180] A third determining unit is used to determine the first k words with the largest attention scores as the target information;

[0181] Wherein, k is the specified window length, which is a positive integer.

[0182] In some embodiments, the first determining unit is further configured to:

[0183] Determine the attention pool memory information and the short-term memory information from the initial context information; the attention pool memory information belongs to the long-term memory information and is located in the first second specific number of words in the first word sequence; the short-term memory information is located in the last first specific number of words in the first word sequence;

[0184] The reasoning module 730 is also used for:

[0185] Based on the attention pool memory information, the target information and the short-term memory information, the large model is used to infer and obtain output information for the information to be processed;

[0186] Among them, the word elements corresponding to the target information are stored in the first storage location in sequence, the word elements corresponding to the short-term memory information are stored in the second storage location in sequence, and the attention pool memory information, the target information and the short-term memory information form the target context information.

[0187] In some embodiments, the reasoning module 730 includes:

[0188] an updating unit, configured to update the target information stored in the first storage location and the short-term memory information stored in the second storage location based on the attention mechanism and the word-units inferred by the large model during the process of performing word-by-word reasoning on the output information using the large model;

[0189] An inference unit is used to continue to infer the next word of the output information using the large model based on the attention pool memory information, the target information stored in the updated first storage location, and the short-term memory information stored in the updated second storage location.

[0190] In some embodiments, the updating unit is further configured to:

[0191] When it is determined that any first target word in the second word sequence corresponding to the output information has been inferred using the large model, the second target word in the word corresponding to the target information currently stored in the first storage position is determined based on the attention mechanism; the second target word is the word with the smallest attention score among the words corresponding to the target information currently stored in the first storage position; after deleting the second target word from the first storage position, the words stored after the second target word in the first storage position are moved forward in sequence; after migrating the first word in the word corresponding to the short-term memory information currently stored in the second storage position to the end position of the first storage position, the words stored after the first word in the second storage position are moved forward in sequence; and the first target word is stored at the end position of the second storage position.

[0192] In some embodiments, the updating unit is further configured to:

[0193] Based on the attention mechanism, the attention scores of each word corresponding to the target information currently stored in the first storage location are determined; based on the binary search method, the second target word with the smallest attention score is determined from the word corresponding to the target information currently stored in the first storage location.

[0194] It should be noted here that the above-mentioned large model reasoning device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned large model reasoning method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0195] Figure 8 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the executable data instructions stored in the memory 830 to execute part or all of the steps in the large model reasoning method provided in the above embodiments.

[0196] In addition, the executable data instructions stored in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0197] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements some or all of the steps in the large model reasoning method provided in the above embodiments.

[0198] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute some or all of the steps in the large model reasoning method provided in the above embodiments.

[0199] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0200] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0201] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] The above description is merely an optional embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A large model reasoning method, comprising: Obtain the long-term memory information needed for the large model to reason about the information to be processed; Sampling the long-term memory information based on an attention mechanism to obtain target information; Based on the target information, the large model is used to infer and obtain output information for the information to be processed.

2. According to the large model reasoning method of claim 1, the step of obtaining the long-term memory information required by the large model to reason about the information to be processed comprises: Obtaining initial context information required by the large model to perform reasoning on the information to be processed; determining the long-term memory information from the initial context information; The long-term memory information is located in the word-grams other than the last first specific number of word-grams in the first word-gram sequence corresponding to the initial context information.

3. According to the large model inference method of claim 2, the sampling of the long-term memory information based on the attention mechanism to obtain the target information comprises: Determining an attention score of a word included in the long-term memory information; Determine the first k words with the largest attention scores as the target information; Wherein, k is the specified window length, which is a positive integer.

4. The large model reasoning method according to claim 2, further comprising: determining attention pool memory information and short-term memory information from the initial context information; The attention pool memory information belongs to the long-term memory information and is located in the first second specific number of words in the first word sequence; the short-term memory information is located in the last first specific number of words in the first word sequence; The step of obtaining output information for the information to be processed by using the large model reasoning based on the target information includes: Based on the attention pool memory information, the target information and the short-term memory information, the large model is used to infer and obtain output information for the information to be processed; Among them, the word elements corresponding to the target information are stored in the first storage location in sequence, the word elements corresponding to the short-term memory information are stored in the second storage location in sequence, and the attention pool memory information, the target information and the short-term memory information form the target context information.

5. The large model reasoning method according to claim 4, wherein the large model reasoning based on the attention pool memory information, the target information and the short-term memory information obtains output information for the information to be processed, comprising: In the process of using the large model to perform word-by-word reasoning on the output information, the target information stored in the first storage location and the short-term memory information stored in the second storage location are updated based on the attention mechanism and the word-units inferred by the large model; Based on the attention pool memory information, the target information stored in the updated first storage location, and the short-term memory information stored in the updated second storage location, the large model is used to continue reasoning about the next word on the output information.

6. The large model inference method according to claim 5, wherein in the process of using the large model to perform word-by-word inference on the output information, updating the target information stored in the first storage location and the short-term memory information stored in the second storage location based on the attention mechanism and the word-by-word inference obtained by the large model, comprises: In a case where it is determined that any first target word in the second word sequence corresponding to the output information has been inferred using the large model, determining a second target word in the word corresponding to the target information currently stored in the first storage location based on the attention mechanism; The second target word-element is the word-element with the smallest attention score among the word-element corresponding to the target information currently stored in the first storage location; After deleting the second target word from the first storage location, the word stored after the second target word in the first storage location is sequentially moved forward; After the first word element in the word element corresponding to the short-term memory information currently stored in the second storage position is moved to the end position of the first storage position, the word elements stored after the first word element in the second storage position are sequentially moved forward; The first target word-gram is stored at the end position of the second storage location.

7. The large model inference method according to claim 6, wherein the step of determining, based on the attention mechanism, a second target word in the word corresponding to the target information currently stored in the first storage location comprises: Based on the attention mechanism, determining an attention score of each word corresponding to the target information currently stored in the first storage location; Based on the binary search method, the second target word-gram with the smallest attention score is determined from the word-grams corresponding to the target information currently stored in the first storage location.

8. A large model reasoning device, comprising: An acquisition module is used to obtain the long-term memory information required by the large model to reason about the information to be processed; A sampling module, used for sampling the long-term memory information based on an attention mechanism to obtain target information; The reasoning module is used to obtain output information for the information to be processed based on the target information and using the large model for reasoning.

9. An electronic device, comprising: A memory for storing executable data instructions; A processor is used to implement the steps in the large model reasoning method described in any one of claims 1 to 7 when executing the executable data instructions stored in the memory.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps in the large model reasoning method described in any one of claims 1 to 7 are implemented.

Images